drashna/ryujinx
1
0
Fork 0
Code Activity Actions

Reduce JIT GC allocations (#2515)

Browse source 
* Turn `MemoryOperand` into a struct

* Remove `IntrinsicOperation`

* Remove `PhiNode`

* Remove `Node`

* Turn `Operand` into a struct

* Turn `Operation` into a struct

* Clean up pool management methods

* Add `Arena` allocator

* Move `OperationHelper` to `Operation.Factory`

* Move `OperandHelper` to `Operand.Factory`

* Optimize `Operation` a bit

* Fix `Arena` initialization

* Rename `NativeList<T>` to `ArenaList<T>`

* Reduce `Operand` size from 88 to 56 bytes

* Reduce `Operation` size from 56 to 40 bytes

* Add optimistic interning of Register & Constant operands

* Optimize `RegisterUsage` pass a bit

* Optimize `RemoveUnusedNodes` pass a bit

Iterating in reverse-order allows killing dependency chains in a single
pass.

* Fix PPTC symbols

* Optimize `BasicBlock` a bit

Reduce allocations from `_successor` & `DominanceFrontiers`

* Fix `Operation` resize

* Make `Arena` expandable

Change the arena allocator to be expandable by allocating in pages, with
some of them being pooled. Currently 32 pages are pooled. An LRU removal
mechanism should probably be added to it.

Apparently MHR can allocate bitmaps large enough to exceed the 16MB
limit for the type.

* Move `Arena` & `ArenaList` to `Common`

* Remove `ThreadStaticPool` & co

* Add `PhiOperation`

* Reduce `Operand` size from 56 from 48 bytes

* Add linear-probing to `Operand` intern table

* Optimize `HybridAllocator` a bit

* Add `Allocators` class

* Tune `ArenaAllocator` sizes

* Add page removal mechanism to `ArenaAllocator`

Remove pages which have not been used for more than 5s after each reset.

I am on fence if this would be better using a Gen2 callback object like
the one in System.Buffers.ArrayPool<T>, to trim the pool. Because right
now if a large translation happens, the pages will be freed only after a
reset. This reset may not happen for a while because no new translation
is hit, but the arena base sizes are rather small.

* Fix `OOM` when allocating larger than page size in `ArenaAllocator`

Tweak resizing mechanism for Operand.Uses and Assignemnts.

* Optimize `Optimizer` a bit

* Optimize `Operand.Add<T>/Remove<T>` a bit

* Clean up `PreAllocator`

* Fix phi insertion order

Reduce codegen diffs.

* Fix code alignment

* Use new heuristics for degree of parallelism

* Suppress warnings

* Address gdkchan's feedback

Renamed `GetValue()` to `GetValueUnsafe()` to make it more clear that
`Operand.Value` should usually not be modified directly.

* Add fast path to `ArenaAllocator`

* Assembly for `ArenaAllocator.Allocate(ulong)`:

  .L0:
    mov rax, [rcx+0x18]
    lea r8, [rax+rdx]
    cmp r8, [rcx+0x10]
    ja short .L2
  .L1:
    mov rdx, [rcx+8]
    add rax, [rdx+8]
    mov [rcx+0x18], r8
    ret
  .L2:
    jmp ArenaAllocator.AllocateSlow(UInt64)

  A few variable/field had to be changed to ulong so that RyuJIT avoids
  emitting zero-extends.

* Implement a new heuristic to free pooled pages.

  If an arena is used often, it is more likely that its pages will be
  needed, so the pages are kept for longer (e.g: during PPTC rebuild or
  burst sof compilations). If is not used often, then it is more likely
  that its pages will not be needed (e.g: after PPTC rebuild or bursts
  of compilations).

* Address riperiperi's feedback

* Use `EqualityComparer<T>` in `IntrusiveList<T>`

Avoids a potential GC hole in `Equals(T, T)`.
This commit is contained in:
FICTURE7 2021-08-17 22:08:34 +04:00 • committed by GitHub
parent cd4530f29c
commit 22b2cb39af
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
91 changed files with 2354 additions and 2142 deletions
Download patch file Download diff file Expand all files Collapse all files

6
ARMeilleure/Translation/ArmEmitterContext.cs
View file

@ -9,7 +9,7 @@ using ARMeilleure.Translation.PTC;
using System;
using System.Collections.Generic;
using System.Reflection;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -138,7 +138,7 @@ namespace ARMeilleure.Translation
{
if (_optOpLastCompare == null || _optOpLastCompare != _optOpLastFlagSet)
{
return null;
return default;
}
Operand n = _optCmpTempN;
@ -193,7 +193,7 @@ namespace ARMeilleure.Translation
}
}
return null;
return default;
}
}
}

2
ARMeilleure/Translation/Compiler.cs
View file

@ -55,7 +55,7 @@ namespace ARMeilleure.Translation
Logger.EndPass(PassName.SsaConstruction, cfg);
CompilerContext cctx = new CompilerContext(cfg, argTypes, retType, options);
CompilerContext cctx = new(cfg, argTypes, retType, options);
return CodeGenerator.Generate(cctx, ptcInfo);
}

19
ARMeilleure/Translation/ControlFlowGraph.cs
View file

@ -22,15 +22,12 @@ namespace ARMeilleure.Translation
Blocks = blocks;
LocalsCount = localsCount;
Update(removeUnreachableBlocks: true);
Update();
}
public void Update(bool removeUnreachableBlocks)
public void Update()
{
if (removeUnreachableBlocks)
{
RemoveUnreachableBlocks(Blocks);
}
RemoveUnreachableBlocks(Blocks);
var visited = new HashSet<BasicBlock>();
var blockStack = new Stack<BasicBlock>();
@ -47,7 +44,7 @@ namespace ARMeilleure.Translation
{
bool visitedNew = false;
for (int i = 0; i < block.SuccessorCount; i++)
for (int i = 0; i < block.SuccessorsCount; i++)
{
BasicBlock succ = block.GetSuccessor(i);
@ -83,7 +80,7 @@ namespace ARMeilleure.Translation
{
Debug.Assert(block.Index != -1, "Invalid block index.");
for (int i = 0; i < block.SuccessorCount; i++)
for (int i = 0; i < block.SuccessorsCount; i++)
{
BasicBlock succ = block.GetSuccessor(i);
@ -105,9 +102,9 @@ namespace ARMeilleure.Translation
if (!visited.Contains(block))
{
while (block.SuccessorCount > 0)
while (block.SuccessorsCount > 0)
{
block.RemoveSuccessor(index: block.SuccessorCount - 1);
block.RemoveSuccessor(index: block.SuccessorsCount - 1);
}
blocks.Remove(block);
@ -126,7 +123,7 @@ namespace ARMeilleure.Translation
{
BasicBlock splitBlock = new BasicBlock(Blocks.Count);
for (int i = 0; i < predecessor.SuccessorCount; i++)
for (int i = 0; i < predecessor.SuccessorsCount; i++)
{
if (predecessor.GetSuccessor(i) == successor)
{

44
ARMeilleure/Translation/EmitterContext.cs
View file

@ -1,12 +1,10 @@
using ARMeilleure.Diagnostics;
using ARMeilleure.IntermediateRepresentation;
using ARMeilleure.State;
using ARMeilleure.Translation.PTC;
using System;
using System.Collections.Generic;
using System.Reflection;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -77,7 +75,7 @@ namespace ARMeilleure.Translation
public void BranchIf(Operand label, Operand op1, Operand op2, Comparison comp, BasicBlockFrequency falseFreq = default)
{
Add(Instruction.BranchIf, null, op1, op2, Const((int)comp));
Add(Instruction.BranchIf, default, op1, op2, Const((int)comp));
BranchToLabel(label, uncond: false, falseFreq);
}
@ -157,7 +155,7 @@ namespace ARMeilleure.Translation
}
else
{
return Add(Instruction.Call, null, args);
return Add(Instruction.Call, default, args);
}
}
@ -169,7 +167,7 @@ namespace ARMeilleure.Translation
Array.Copy(callArgs, 0, args, 1, callArgs.Length);
Add(Instruction.Tailcall, null, args);
Add(Instruction.Tailcall, default, args);
_needsNewBlock = true;
}
@ -356,7 +354,7 @@ namespace ARMeilleure.Translation
public void Return(Operand op1)
{
Add(Instruction.Return, null, op1);
Add(Instruction.Return, default, op1);
_needsNewBlock = true;
}
@ -398,17 +396,17 @@ namespace ARMeilleure.Translation
public void Store(Operand address, Operand value)
{
Add(Instruction.Store, null, address, value);
Add(Instruction.Store, default, address, value);
}
public void Store16(Operand address, Operand value)
{
Add(Instruction.Store16, null, address, value);
Add(Instruction.Store16, default, address, value);
}
public void Store8(Operand address, Operand value)
{
Add(Instruction.Store8, null, address, value);
Add(Instruction.Store8, default, address, value);
}
public void StoreToContext()
@ -501,11 +499,11 @@ namespace ARMeilleure.Translation
}
}
private Operand Add(Instruction inst, Operand dest = null)
private Operand Add(Instruction inst, Operand dest = default)
{
NewNextBlockIfNeeded();
Operation operation = OperationHelper.Operation(inst, dest);
Operation operation = Operation.Factory.Operation(inst, dest);
_irBlock.Operations.AddLast(operation);
@ -516,7 +514,7 @@ namespace ARMeilleure.Translation
{
NewNextBlockIfNeeded();
Operation operation = OperationHelper.Operation(inst, dest, sources);
Operation operation = Operation.Factory.Operation(inst, dest, sources);
_irBlock.Operations.AddLast(operation);
@ -527,7 +525,7 @@ namespace ARMeilleure.Translation
{
NewNextBlockIfNeeded();
Operation operation = OperationHelper.Operation(inst, dest, source0);
Operation operation = Operation.Factory.Operation(inst, dest, source0);
_irBlock.Operations.AddLast(operation);
@ -538,7 +536,7 @@ namespace ARMeilleure.Translation
{
NewNextBlockIfNeeded();
Operation operation = OperationHelper.Operation(inst, dest, source0, source1);
Operation operation = Operation.Factory.Operation(inst, dest, source0, source1);
_irBlock.Operations.AddLast(operation);
@ -549,7 +547,7 @@ namespace ARMeilleure.Translation
{
NewNextBlockIfNeeded();
Operation operation = OperationHelper.Operation(inst, dest, source0, source1, source2);
Operation operation = Operation.Factory.Operation(inst, dest, source0, source1, source2);
_irBlock.Operations.AddLast(operation);
@ -573,14 +571,14 @@ namespace ARMeilleure.Translation
public void AddIntrinsicNoRet(Intrinsic intrin, params Operand[] args)
{
Add(intrin, null, args);
Add(intrin, default, args);
}
private Operand Add(Intrinsic intrin, Operand dest, params Operand[] sources)
{
NewNextBlockIfNeeded();
IntrinsicOperation operation = new IntrinsicOperation(intrin, dest, sources);
Operation operation = Operation.Factory.Operation(intrin, dest, sources);
_irBlock.Operations.AddLast(operation);
@ -641,7 +639,7 @@ namespace ARMeilleure.Translation
private void NextBlock(BasicBlock nextBlock)
{
if (_irBlock?.SuccessorCount == 0 && !EndsWithUnconditional(_irBlock))
if (_irBlock?.SuccessorsCount == 0 && !EndsWithUnconditional(_irBlock))
{
_irBlock.AddSuccessor(nextBlock);
@ -662,9 +660,11 @@ namespace ARMeilleure.Translation
private static bool EndsWithUnconditional(BasicBlock block)
{
return block.Operations.Last is Operation lastOp &&
(lastOp.Instruction == Instruction.Return ||
lastOp.Instruction == Instruction.Tailcall);
Operation last = block.Operations.Last;
return last != default &&
(last.Instruction == Instruction.Return ||
last.Instruction == Instruction.Tailcall);
}
public ControlFlowGraph GetControlFlowGraph()

50
ARMeilleure/Translation/PTC/DegreeOfParallelism.cs
View file

@ -1,50 +0,0 @@
using System;
namespace ARMeilleure.Translation.PTC
{
class DegreeOfParallelism
{
public double GiBRef { get; } // GiB.
public double WeightRef { get; } // %.
public double IncrementByGiB { get; } // %.
private double _coefficient;
public DegreeOfParallelism(double gibRef, double weightRef, double incrementByGiB)
{
GiBRef = gibRef;
WeightRef = weightRef;
IncrementByGiB = incrementByGiB;
_coefficient = weightRef - (incrementByGiB * gibRef);
}
public int GetDegreeOfParallelism(int min, int max)
{
double degreeOfParallelism = (GetProcessorCount() * GetWeight(GetAvailableMemoryGiB())) / 100d;
return Math.Clamp((int)Math.Round(degreeOfParallelism), min, max);
}
public static double GetProcessorCount()
{
return (double)Environment.ProcessorCount;
}
public double GetWeight(double gib)
{
return (IncrementByGiB * gib) + _coefficient;
}
public static double GetAvailableMemoryGiB()
{
GCMemoryInfo gcMemoryInfo = GC.GetGCMemoryInfo();
return FromBytesToGiB(gcMemoryInfo.TotalAvailableMemoryBytes - gcMemoryInfo.MemoryLoadBytes);
}
private static double FromBytesToGiB(long bytes)
{
return Math.ScaleB((double)bytes, -30);
}
}
}

23
ARMeilleure/Translation/PTC/Ptc.cs
View file

@ -9,7 +9,6 @@ using Ryujinx.Common.Configuration;
using Ryujinx.Common.Logging;
using System;
using System.Buffers.Binary;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
@ -28,7 +27,7 @@ namespace ARMeilleure.Translation.PTC
private const string OuterHeaderMagicString = "PTCohd\0\0";
private const string InnerHeaderMagicString = "PTCihd\0\0";
private const uint InternalVersion = 2228; //! To be incremented manually for each change to the ARMeilleure project.
private const uint InternalVersion = 2515; //! To be incremented manually for each change to the ARMeilleure project.
private const string ActualDir = "0";
private const string BackupDir = "1";
@ -776,9 +775,7 @@ namespace ARMeilleure.Translation.PTC
_translateCount = 0;
_translateTotalCount = profiledFuncsToTranslate.Count;
int degreeOfParallelism = new DegreeOfParallelism(4d, 75d, 12.5d).GetDegreeOfParallelism(0, 32);
if (_translateTotalCount == 0 || degreeOfParallelism == 0)
if (_translateTotalCount == 0)
{
ResetCarriersIfNeeded();
@ -787,6 +784,14 @@ namespace ARMeilleure.Translation.PTC
return;
}
int degreeOfParallelism = Environment.ProcessorCount;
// If there are enough cores lying around, we leave one alone for other tasks.
if (degreeOfParallelism > 4)
{
degreeOfParallelism--;
}
Logger.Info?.Print(LogClass.Ptc, $"{_translateCount} of {_translateTotalCount} functions translated | Thread count: {degreeOfParallelism}");
PtcStateChanged?.Invoke(PtcLoadingState.Start, _translateCount, _translateTotalCount);
@ -825,8 +830,6 @@ namespace ARMeilleure.Translation.PTC
break;
}
}
Translator.DisposePools();
}
List<Thread> threads = new List<Thread>();
@ -839,6 +842,8 @@ namespace ARMeilleure.Translation.PTC
threads.Add(thread);
}
Stopwatch sw = Stopwatch.StartNew();
threads.ForEach((thread) => thread.Start());
threads.ForEach((thread) => thread.Join());
@ -847,9 +852,11 @@ namespace ARMeilleure.Translation.PTC
progressReportEvent.Set();
progressReportThread.Join();
sw.Stop();
PtcStateChanged?.Invoke(PtcLoadingState.Loaded, _translateCount, _translateTotalCount);
Logger.Info?.Print(LogClass.Ptc, $"{_translateCount} of {_translateTotalCount} functions translated | Thread count: {degreeOfParallelism}");
Logger.Info?.Print(LogClass.Ptc, $"{_translateCount} of {_translateTotalCount} functions translated | Thread count: {degreeOfParallelism} in {sw.Elapsed.TotalSeconds} s");
Thread preSaveThread = new Thread(PreSave);
preSaveThread.IsBackground = true;

6
ARMeilleure/Translation/RegisterToLocal.cs
View file

@ -1,7 +1,7 @@
using ARMeilleure.IntermediateRepresentation;
using System.Collections.Generic;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -27,11 +27,11 @@ namespace ARMeilleure.Translation
for (BasicBlock block = cfg.Blocks.First; block != null; block = block.ListNext)
{
for (Node node = block.Operations.First; node != null; node = node.ListNext)
for (Operation node = block.Operations.First; node != default; node = node.ListNext)
{
Operand dest = node.Destination;
if (dest != null && dest.Kind == OperandKind.Register)
if (dest != default && dest.Kind == OperandKind.Register)
{
node.Destination = GetLocal(dest);
}

220
ARMeilleure/Translation/RegisterUsage.cs
View file

@ -1,9 +1,11 @@
using ARMeilleure.IntermediateRepresentation;
using ARMeilleure.State;
using System;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.OperationHelper;
using System.Numerics;
using System.Runtime.Intrinsics;
using System.Runtime.Intrinsics.X86;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
using static ARMeilleure.IntermediateRepresentation.Operation.Factory;
namespace ARMeilleure.Translation
{
@ -14,27 +16,48 @@ namespace ARMeilleure.Translation
private struct RegisterMask : IEquatable<RegisterMask>
{
public long IntMask { get; set; }
public long VecMask { get; set; }
public long IntMask => Mask.GetElement(0);
public long VecMask => Mask.GetElement(1);
public Vector128<long> Mask { get; }
public RegisterMask(Vector128<long> mask)
{
Mask = mask;
}
public RegisterMask(long intMask, long vecMask)
{
IntMask = intMask;
VecMask = vecMask;
Mask = Vector128.Create(intMask, vecMask);
}
public static RegisterMask operator &(RegisterMask x, RegisterMask y)
{
if (Sse2.IsSupported)
{
return new RegisterMask(Sse2.And(x.Mask, y.Mask));
}
return new RegisterMask(x.IntMask & y.IntMask, x.VecMask & y.VecMask);
}
public static RegisterMask operator |(RegisterMask x, RegisterMask y)
{
if (Sse2.IsSupported)
{
return new RegisterMask(Sse2.Or(x.Mask, y.Mask));
}
return new RegisterMask(x.IntMask | y.IntMask, x.VecMask | y.VecMask);
}
public static RegisterMask operator ~(RegisterMask x)
{
if (Sse2.IsSupported)
{
return new RegisterMask(Sse2.AndNot(x.Mask, Vector128<long>.AllBitsSet));
}
return new RegisterMask(~x.IntMask, ~x.VecMask);
}
@ -55,12 +78,12 @@ namespace ARMeilleure.Translation
public bool Equals(RegisterMask other)
{
return IntMask == other.IntMask && VecMask == other.VecMask;
return Mask.Equals(other.Mask);
}
public override int GetHashCode()
{
return HashCode.Combine(IntMask, VecMask);
return Mask.GetHashCode();
}
}
@ -72,27 +95,23 @@ namespace ARMeilleure.Translation
for (BasicBlock block = cfg.Blocks.First; block != null; block = block.ListNext)
{
for (Node node = block.Operations.First; node != null; node = node.ListNext)
for (Operation node = block.Operations.First; node != default; node = node.ListNext)
{
Operation operation = node as Operation;
for (int srcIndex = 0; srcIndex < operation.SourcesCount; srcIndex++)
for (int index = 0; index < node.SourcesCount; index++)
{
Operand source = operation.GetSource(srcIndex);
Operand source = node.GetSource(index);
if (source.Kind != OperandKind.Register)
if (source.Kind == OperandKind.Register)
{
continue;
Register register = source.GetRegister();
localInputs[block.Index] |= GetMask(register) & ~localOutputs[block.Index];
}
Register register = source.GetRegister();
localInputs[block.Index] |= GetMask(register) & ~localOutputs[block.Index];
}
if (operation.Destination != null && operation.Destination.Kind == OperandKind.Register)
if (node.Destination != default && node.Destination.Kind == OperandKind.Register)
{
localOutputs[block.Index] |= GetMask(operation.Destination.GetRegister());
localOutputs[block.Index] |= GetMask(node.Destination.GetRegister());
}
}
}
@ -104,7 +123,6 @@ namespace ARMeilleure.Translation
RegisterMask[] globalOutputs = new RegisterMask[cfg.Blocks.Count];
bool modified;
bool firstPass = true;
do
@ -121,7 +139,6 @@ namespace ARMeilleure.Translation
BasicBlock predecessor = block.Predecessors[0];
RegisterMask cmnOutputs = localOutputs[predecessor.Index] | globalCmnOutputs[predecessor.Index];
RegisterMask outputs = globalOutputs[predecessor.Index];
for (int pIndex = 1; pIndex < block.Predecessors.Count; pIndex++)
@ -129,7 +146,6 @@ namespace ARMeilleure.Translation
predecessor = block.Predecessors[pIndex];
cmnOutputs &= localOutputs[predecessor.Index] | globalCmnOutputs[predecessor.Index];
outputs |= globalOutputs[predecessor.Index];
}
@ -140,21 +156,13 @@ namespace ARMeilleure.Translation
cmnOutputs &= globalCmnOutputs[block.Index];
}
if (Exchange(globalCmnOutputs, block.Index, cmnOutputs))
{
modified = true;
}
modified |= Exchange(globalCmnOutputs, block.Index, cmnOutputs);
outputs |= localOutputs[block.Index];
if (Exchange(globalOutputs, block.Index, globalOutputs[block.Index] | outputs))
{
modified = true;
}
modified |= Exchange(globalOutputs, block.Index, globalOutputs[block.Index] | outputs);
}
else if (Exchange(globalOutputs, block.Index, localOutputs[block.Index]))
else
{
modified = true;
modified |= Exchange(globalOutputs, block.Index, localOutputs[block.Index]);
}
}
@ -165,17 +173,14 @@ namespace ARMeilleure.Translation
RegisterMask inputs = localInputs[block.Index];
for (int i = 0; i < block.SuccessorCount; i++)
for (int i = 0; i < block.SuccessorsCount; i++)
{
inputs |= globalInputs[block.GetSuccessor(i).Index];
}
inputs &= ~globalCmnOutputs[block.Index];
if (Exchange(globalInputs, block.Index, globalInputs[block.Index] | inputs))
{
modified = true;
}
modified |= Exchange(globalInputs, block.Index, globalInputs[block.Index] | inputs);
}
firstPass = false;
@ -192,12 +197,18 @@ namespace ARMeilleure.Translation
block.Operations.Remove(block.Operations.First);
}
Operand arg = default;
// The only block without any predecessor should be the entry block.
// It always needs a context load as it is the first block to run.
if (block.Predecessors.Count == 0 || hasContextLoad)
{
LoadLocals(block, globalInputs[block.Index].VecMask, RegisterType.Vector, mode);
LoadLocals(block, globalInputs[block.Index].IntMask, RegisterType.Integer, mode);
arg = Local(OperandType.I64);
Operation loadArg = block.Operations.AddFirst(Operation(Instruction.LoadArgument, arg, Const(0)));
LoadLocals(block, globalInputs[block.Index].VecMask, RegisterType.Vector, mode, loadArg, arg);
LoadLocals(block, globalInputs[block.Index].IntMask, RegisterType.Integer, mode, loadArg, arg);
}
bool hasContextStore = HasContextStore(block);
@ -209,8 +220,15 @@ namespace ARMeilleure.Translation
if (EndsWithReturn(block) || hasContextStore)
{
StoreLocals(block, globalOutputs[block.Index].IntMask, RegisterType.Integer, mode);
StoreLocals(block, globalOutputs[block.Index].VecMask, RegisterType.Vector, mode);
if (arg == default)
{
arg = Local(OperandType.I64);
block.Append(Operation(Instruction.LoadArgument, arg, Const(0)));
}
StoreLocals(block, globalOutputs[block.Index].IntMask, RegisterType.Integer, mode, arg);
StoreLocals(block, globalOutputs[block.Index].VecMask, RegisterType.Vector, mode, arg);
}
}
}
@ -222,27 +240,31 @@ namespace ARMeilleure.Translation
private static bool HasContextStore(BasicBlock block)
{
return EndsWith(block, Instruction.StoreToContext) && block.GetLastOp().SourcesCount == 0;
return EndsWith(block, Instruction.StoreToContext) && block.Operations.Last.SourcesCount == 0;
}
private static bool StartsWith(BasicBlock block, Instruction inst)
{
if (block.Operations.Count == 0)
if (block.Operations.Count > 0)
{
return false;
Operation first = block.Operations.First;
return first != default && first.Instruction == inst;
}
return block.Operations.First is Operation operation && operation.Instruction == inst;
return false;
}
private static bool EndsWith(BasicBlock block, Instruction inst)
{
if (block.Operations.Count == 0)
if (block.Operations.Count > 0)
{
return false;
Operation last = block.Operations.Last;
return last != default && last.Instruction == inst;
}
return block.Operations.Last is Operation operation && operation.Instruction == inst;
return false;
}
private static RegisterMask GetMask(Register register)
@ -263,76 +285,57 @@ namespace ARMeilleure.Translation
private static bool Exchange(RegisterMask[] masks, int blkIndex, RegisterMask value)
{
RegisterMask oldValue = masks[blkIndex];
ref RegisterMask curValue = ref masks[blkIndex];
masks[blkIndex] = value;
bool changed = curValue != value;
return oldValue != value;
curValue = value;
return changed;
}
private static void LoadLocals(BasicBlock block, long inputs, RegisterType baseType, ExecutionMode mode)
private static void LoadLocals(
BasicBlock block,
long inputs,
RegisterType baseType,
ExecutionMode mode,
Operation loadArg,
Operand arg)
{
Operand arg0 = Local(OperandType.I64);
for (int bit = 63; bit >= 0; bit--)
while (inputs != 0)
{
long mask = 1L << bit;
if ((inputs & mask) == 0)
{
continue;
}
int bit = 63 - BitOperations.LeadingZeroCount((ulong)inputs);
Operand dest = GetRegFromBit(bit, baseType, mode);
long offset = NativeContext.GetRegisterOffset(dest.GetRegister());
Operand offset = Const((long)NativeContext.GetRegisterOffset(dest.GetRegister()));
Operand addr = Local(OperandType.I64);
Operation loadOp = Operation(Instruction.Load, dest, addr);
block.Operations.AddAfter(loadArg, Operation(Instruction.Load, dest, addr));
block.Operations.AddAfter(loadArg, Operation(Instruction.Add, addr, arg, offset));
block.Operations.AddFirst(loadOp);
Operation calcOffsOp = Operation(Instruction.Add, addr, arg0, Const(offset));
block.Operations.AddFirst(calcOffsOp);
inputs &= ~(1L << bit);
}
Operation loadArg0 = Operation(Instruction.LoadArgument, arg0, Const(0));
block.Operations.AddFirst(loadArg0);
}
private static void StoreLocals(BasicBlock block, long outputs, RegisterType baseType, ExecutionMode mode)
private static void StoreLocals(
BasicBlock block,
long outputs,
RegisterType baseType,
ExecutionMode mode,
Operand arg)
{
Operand arg0 = Local(OperandType.I64);
Operation loadArg0 = Operation(Instruction.LoadArgument, arg0, Const(0));
block.Append(loadArg0);
for (int bit = 0; bit < 64; bit++)
while (outputs != 0)
{
long mask = 1L << bit;
if ((outputs & mask) == 0)
{
continue;
}
int bit = BitOperations.TrailingZeroCount(outputs);
Operand source = GetRegFromBit(bit, baseType, mode);
long offset = NativeContext.GetRegisterOffset(source.GetRegister());
Operand offset = Const((long)NativeContext.GetRegisterOffset(source.GetRegister()));
Operand addr = Local(OperandType.I64);
Operation calcOffsOp = Operation(Instruction.Add, addr, arg0, Const(offset));
block.Append(Operation(Instruction.Add, addr, arg, offset));
block.Append(Operation(Instruction.Store, default, addr, source));
block.Append(calcOffsOp);
Operation storeOp = Operation(Instruction.Store, null, addr, source);
block.Append(storeOp);
outputs &= ~(1L << bit);
}
}
@ -340,15 +343,15 @@ namespace ARMeilleure.Translation
{
if (bit < RegsCount)
{
return OperandHelper.Register(bit, baseType, GetOperandType(baseType, mode));
return Register(bit, baseType, GetOperandType(baseType, mode));
}
else if (baseType == RegisterType.Integer)
{
return OperandHelper.Register(bit & RegsMask, RegisterType.Flag, OperandType.I32);
return Register(bit & RegsMask, RegisterType.Flag, OperandType.I32);
}
else if (baseType == RegisterType.Vector)
{
return OperandHelper.Register(bit & RegsMask, RegisterType.FpFlag, OperandType.I32);
return Register(bit & RegsMask, RegisterType.FpFlag, OperandType.I32);
}
else
{
@ -371,12 +374,9 @@ namespace ARMeilleure.Translation
private static bool EndsWithReturn(BasicBlock block)
{
if (!(block.GetLastOp() is Operation operation))
{
return false;
}
Operation last = block.Operations.Last;
return operation.Instruction == Instruction.Return;
return last != default && last.Instruction == Instruction.Return;
}
}
}

112
ARMeilleure/Translation/SsaConstruction.cs
View file

@ -4,7 +4,7 @@ using ARMeilleure.State;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -18,7 +18,7 @@ namespace ARMeilleure.Translation
public DefMap()
{
_map = new Dictionary<int, Operand>();
_phiMasks = new BitMap(RegisterConsts.TotalCount);
_phiMasks = new BitMap(Allocators.Default, RegisterConsts.TotalCount);
}
public bool TryAddOperand(int key, Operand operand)
@ -57,26 +57,26 @@ namespace ARMeilleure.Translation
// First pass, get all defs and locals uses.
for (BasicBlock block = cfg.Blocks.First; block != null; block = block.ListNext)
{
for (Node node = block.Operations.First; node != null; node = node.ListNext)
for (Operation node = block.Operations.First; node != default; node = node.ListNext)
{
if (node is not Operation operation)
for (int index = 0; index < node.SourcesCount; index++)
{
continue;
}
for (int index = 0; index < operation.SourcesCount; index++)
{
Operand src = operation.GetSource(index);
Operand src = node.GetSource(index);
if (TryGetId(src, out int srcKey))
{
Operand local = localDefs[srcKey] ?? src;
Operand local = localDefs[srcKey];
operation.SetSource(index, local);
if (local == default)
{
local = src;
}
node.SetSource(index, local);
}
}
Operand dest = operation.Destination;
Operand dest = node.Destination;
if (TryGetId(dest, out int destKey))
{
@ -84,7 +84,7 @@ namespace ARMeilleure.Translation
localDefs[destKey] = local;
operation.Destination = local;
node.Destination = local;
}
}
@ -92,7 +92,7 @@ namespace ARMeilleure.Translation
{
Operand local = localDefs[key];
if (local is null)
if (local == default)
{
continue;
}
@ -119,28 +119,23 @@ namespace ARMeilleure.Translation
// Second pass, rename variables with definitions on different blocks.
for (BasicBlock block = cfg.Blocks.First; block != null; block = block.ListNext)
{
for (Node node = block.Operations.First; node != null; node = node.ListNext)
for (Operation node = block.Operations.First; node != default; node = node.ListNext)
{
if (node is not Operation operation)
for (int index = 0; index < node.SourcesCount; index++)
{
continue;
}
for (int index = 0; index < operation.SourcesCount; index++)
{
Operand src = operation.GetSource(index);
Operand src = node.GetSource(index);
if (TryGetId(src, out int key))
{
Operand local = localDefs[key];
if (local is null)
if (local == default)
{
local = FindDef(globalDefs, block, src);
localDefs[key] = local;
}
operation.SetSource(index, local);
node.SetSource(index, local);
}
}
}
@ -200,12 +195,14 @@ namespace ARMeilleure.Translation
// then use the definition from that Phi.
Operand local = Local(operand.Type);
PhiNode phi = new PhiNode(local, block.Predecessors.Count);
Operation operation = Operation.Factory.PhiOperation(local, block.Predecessors.Count);
AddPhi(block, phi);
AddPhi(block, operation);
globalDefs[block.Index].TryAddOperand(GetId(operand), local);
PhiOperation phi = operation.AsPhi();
for (int index = 0; index < block.Predecessors.Count; index++)
{
BasicBlock predecessor = block.Predecessors[index];
@ -217,19 +214,19 @@ namespace ARMeilleure.Translation
return local;
}
private static void AddPhi(BasicBlock block, PhiNode phi)
private static void AddPhi(BasicBlock block, Operation phi)
{
Node node = block.Operations.First;
Operation node = block.Operations.First;
if (node != null)
if (node != default)
{
while (node.ListNext is PhiNode)
while (node.ListNext != default && node.ListNext.Instruction == Instruction.Phi)
{
node = node.ListNext;
}
}
if (node is PhiNode)
if (node != default && node.Instruction == Instruction.Phi)
{
block.Operations.AddAfter(node, phi);
}
@ -241,34 +238,37 @@ namespace ARMeilleure.Translation
private static bool TryGetId(Operand operand, out int result)
{
if (operand is { Kind: OperandKind.Register })
if (operand != default)
{
Register reg = operand.GetRegister();
if (operand.Kind == OperandKind.Register)
{
Register reg = operand.GetRegister();
if (reg.Type == RegisterType.Integer)
{
result = reg.Index;
}
else if (reg.Type == RegisterType.Vector)
{
result = RegisterConsts.IntRegsCount + reg.Index;
}
else if (reg.Type == RegisterType.Flag)
{
result = RegisterConsts.IntAndVecRegsCount + reg.Index;
}
else /* if (reg.Type == RegisterType.FpFlag) */
{
result = RegisterConsts.FpFlagsOffset + reg.Index;
}
if (reg.Type == RegisterType.Integer)
{
result = reg.Index;
}
else if (reg.Type == RegisterType.Vector)
{
result = RegisterConsts.IntRegsCount + reg.Index;
}
else if (reg.Type == RegisterType.Flag)
{
result = RegisterConsts.IntAndVecRegsCount + reg.Index;
}
else /* if (reg.Type == RegisterType.FpFlag) */
{
result = RegisterConsts.FpFlagsOffset + reg.Index;
}
return true;
}
else if (operand is { Kind: OperandKind.LocalVariable } && operand.GetLocalNumber() > 0)
{
result = RegisterConsts.TotalCount + operand.GetLocalNumber() - 1;
return true;
}
else if (operand.Kind == OperandKind.LocalVariable && operand.GetLocalNumber() > 0)
{
result = RegisterConsts.TotalCount + operand.GetLocalNumber() - 1;
return true;
return true;
}
}
result = -1;

28
ARMeilleure/Translation/SsaDeconstruction.cs
View file

@ -1,7 +1,7 @@
using ARMeilleure.IntermediateRepresentation;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.OperationHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
using static ARMeilleure.IntermediateRepresentation.Operation.Factory;
namespace ARMeilleure.Translation
{
@ -11,34 +11,36 @@ namespace ARMeilleure.Translation
{
for (BasicBlock block = cfg.Blocks.First; block != null; block = block.ListNext)
{
Node node = block.Operations.First;
Operation operation = block.Operations.First;
while (node is PhiNode phi)
while (operation != default && operation.Instruction == Instruction.Phi)
{
Node nextNode = node.ListNext;
Operation nextNode = operation.ListNext;
Operand local = Local(phi.Destination.Type);
Operand local = Local(operation.Destination.Type);
PhiOperation phi = operation.AsPhi();
for (int index = 0; index < phi.SourcesCount; index++)
{
BasicBlock predecessor = phi.GetBlock(index);
BasicBlock predecessor = phi.GetBlock(cfg, index);
Operand source = phi.GetSource(index);
predecessor.Append(Operation(Instruction.Copy, local, source));
phi.SetSource(index, null);
phi.SetSource(index, default);
}
Operation copyOp = Operation(Instruction.Copy, phi.Destination, local);
Operation copyOp = Operation(Instruction.Copy, operation.Destination, local);
block.Operations.AddBefore(node, copyOp);
block.Operations.AddBefore(operation, copyOp);
phi.Destination = null;
operation.Destination = default;
block.Operations.Remove(node);
block.Operations.Remove(operation);
node = nextNode;
operation = nextNode;
}
}
}

43
ARMeilleure/Translation/Translator.cs
View file

@ -13,12 +13,8 @@ using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Diagnostics;
using System.Runtime;
using System.Threading;
using static ARMeilleure.Common.BitMapPool;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.OperationHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -193,13 +189,9 @@ namespace ARMeilleure.Translation
ClearJitCache();
DisposePools();
Stubs.Dispose();
FunctionTable.Dispose();
CountTable.Dispose();
GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
}
}
@ -266,8 +258,6 @@ namespace ARMeilleure.Translation
Logger.EndPass(PassName.Decoding);
PreparePool(highCq ? 1 : 0);
Logger.StartPass(PassName.Translation);
EmitSynchronization(context);
@ -281,7 +271,7 @@ namespace ARMeilleure.Translation
ulong funcSize = funcRange.End - funcRange.Start;
Logger.EndPass(PassName.Translation);
Logger.EndPass(PassName.Translation, cfg);
Logger.StartPass(PassName.RegisterUsage);
@ -298,8 +288,6 @@ namespace ARMeilleure.Translation
if (!context.HasPtc)
{
func = Compiler.Compile<GuestFunction>(cfg, argTypes, OperandType.I64, options);
ResetPool(highCq ? 1 : 0);
}
else
{
@ -307,33 +295,16 @@ namespace ARMeilleure.Translation
func = Compiler.Compile<GuestFunction>(cfg, argTypes, OperandType.I64, options, ptcInfo);
ResetPool(highCq ? 1 : 0);
Hash128 hash = Ptc.ComputeHash(Memory, address, funcSize);
Ptc.WriteInfoCodeRelocUnwindInfo(address, funcSize, hash, highCq, ptcInfo);
}
return new TranslatedFunction(func, counter, funcSize, highCq);
}
var result = new TranslatedFunction(func, counter, funcSize, highCq);
internal static void PreparePool(int groupId = 0)
{
PrepareOperandPool(groupId);
PrepareOperationPool(groupId);
}
Allocators.ResetAll();
internal static void ResetPool(int groupId = 0)
{
ResetOperationPool(groupId);
ResetOperandPool(groupId);
}
internal static void DisposePools()
{
DisposeOperandPools();
DisposeOperationPools();
DisposeBitMapPools();
return result;
}
private struct Range
@ -408,7 +379,7 @@ namespace ARMeilleure.Translation
EmitSynchronization(context);
}
Operand lblPredicateSkip = null;
Operand lblPredicateSkip = default;
if (opCode is OpCode32 op && op.Cond < Condition.Al)
{
@ -426,7 +397,7 @@ namespace ARMeilleure.Translation
throw new InvalidOperationException($"Invalid instruction \"{opCode.Instruction.Name}\".");
}
if (lblPredicateSkip != null)
if (lblPredicateSkip != default)
{
context.MarkLabel(lblPredicateSkip);
}

4
ARMeilleure/Translation/TranslatorStubs.cs
View file

@ -5,7 +5,7 @@ using ARMeilleure.Translation.Cache;
using System;
using System.Reflection;
using System.Runtime.InteropServices;
using static ARMeilleure.IntermediateRepresentation.OperandHelper;
using static ARMeilleure.IntermediateRepresentation.Operand.Factory;
namespace ARMeilleure.Translation
{
@ -145,7 +145,7 @@ namespace ARMeilleure.Translation
Operand masked = context.BitwiseAnd(guestAddress, Const(~_translator.FunctionTable.Mask));
context.BranchIfTrue(lblFallback, masked);
Operand index = null;
Operand index = default;
Operand page = Const((long)_translator.FunctionTable.Base);
for (int i = 0; i < _translator.FunctionTable.Levels.Length; i++)