KeyNotFoundException: The given key 'IsDeleted' was not present in the dictionary.

[LEIRONGHUA]

Description

When using Z.EntityFramework.Extensions.EFCore with Volo.Abp.EntityFrameworkCore, a KeyNotFoundException: The given key 'IsDeleted' was not present in the dictionary is thrown when calling BulkInsertAsync with a custom options.ColumnPrimaryKeyExpression that explicitly includes the IsDeleted soft-delete property

This issue is database-agnostic and occurs consistently on both PostgreSQL and SQL Server database providers.

Code Sample

await context.BulkInsertAsync(entities, options =>
{
    // Explicitly include IsDeleted in composite primary key
    options.ColumnPrimaryKeyExpression = p => new
    {
        p.Name,
        p.IsDeleted // ABP soft-delete property
    };
});

Environment

• Library: Z.EntityFramework.Extensions.EFCore v10.105.5
• Framework: Volo.Abp.EntityFrameworkCore (ABP Framework) v10.4.0
• Database Providers: PostgreSQL, SQL Server (the issue occurs in both databases)
• Exception Type: System.Collections.Generic.KeyNotFoundException

Exception Details

Unhandled exception. System.AggregateException: One or more errors occurred. (The given key 'IsDeleted' was not present in the dictionary.)
 ---> System.Collections.Generic.KeyNotFoundException: The given key 'IsDeleted' was not present in the dictionary.
   at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
   at .`1.( , String )
   at .GetValue( accessor)
   at .GetValue( accessor)
   at .( , DbCommand )
   at .( , DbCommand )
   at .( , DbCommand )
   at .( , DbCommand )
   at .Execute(List`1 actions)
   at .(List`1 )
   at Z.BulkOperations.BulkOperation.Execute()
   at Z.BulkOperations.BulkOperation.BulkInsert()
   at .BulkInsert[T](DbContext this, IEntityType entityType, IEnumerable`1 list, Action`1 options, SavingSelector savingSelector, Boolean forceSpecificTypeMapping, Boolean isOptimized)
   at .BulkInsert[T](DbContext this, IEnumerable`1 entities, BulkOperation`1 optionsInstance, Action`1 options, Boolean isBulkSaveChanges, Boolean isOptimized)
   at DbContextExtensions.BulkInsert[T](DbContext this, IEnumerable`1 entities, Action`1 options)
   at DbContextExtensions.`1.()
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at .[](DbContext , Func`2 , Action`1 , CancellationToken )
   at Z.EfCode.Extensions.PgBulkInsert.Program.MainAsync() in C:\Users\UserName\RiderProjects\Z.EfCode.Extensions\Z.EfCode.Extensions.PgBulkInsert\Program.cs:line 45
   at Z.EfCode.Extensions.PgBulkInsert.Program.MainAsync() in C:\Users\UserName\RiderProjects\Z.EfCode.Extensions\Z.EfCode.Extensions.PgBulkInsert\Program.cs:line 79
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at Z.EfCode.Extensions.PgBulkInsert.Program.Main() 

Steps to Reproduce

1. Create a project using Volo.Abp.EntityFrameworkCore with entities implementing ABP's ISoftDelete interface (containing the IsDeleted property).
2. Install and configure Z.EntityFramework.Extensions.EFCore.
3. Configure the project to use either PostgreSQL or SQL Server database provider.
4. Execute any BulkInsertAsync operation
5. The KeyNotFoundException for the IsDeleted key is thrown immediately.

Minimal Repro Code

This is a self-contained, minimal console application that reproduces the error.

1. project.csproj

<Project Sdk="Microsoft.NET.Sdk">

    <PropertyGroup>
        <OutputType>Exe</OutputType>
        <TargetFramework>net10.0</TargetFramework>
        <ImplicitUsings>enable</ImplicitUsings>
        <Nullable>enable</Nullable>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="10.0.8"/>
        <PackageReference Include="Volo.Abp.Ddd.Domain" Version="10.4.0"/>
        <PackageReference Include="Volo.Abp.EntityFrameworkCore.PostgreSql" Version="10.4.0"/>
        <PackageReference Include="Volo.Abp.EntityFrameworkCore.SqlServer" Version="10.4.0"/>
        <PackageReference Include="Z.EntityFramework.Extensions.EFCore" Version="10.105.5"/>
    </ItemGroup>

</Project>

2. Program.cs

using System.Diagnostics;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.Logging;
using Npgsql;
using Volo.Abp.Domain.Entities.Auditing;
using Volo.Abp.EntityFrameworkCore.Modeling;
using Z.BulkOperations;

public class Program
{
    private static readonly ILoggerFactory LoggerFactory = Microsoft.Extensions.Logging.LoggerFactory.Create(builder =>
    {
        builder.AddConsole();
        builder.SetMinimumLevel(LogLevel.Information);
    });

    private static readonly ILogger<Program> Logger = LoggerFactory.CreateLogger<Program>();

    private static async Task<int> Main(string[] args)
    {
        await MainAsync();
        return 0;
    }

    private static async Task MainAsync()
    {
        await using (var context = new EntityContext())
        {
            await context.Database.EnsureCreatedAsync();
        }

        var customers = GenerateCustomers(5);

        var auditEntries = new List<AuditEntry>();
        await using (var context = new EntityContext())
        {
            var tcs = new CancellationTokenSource();
            var token = tcs.Token;

            var stopwatch = Stopwatch.StartNew();

            Logger.LogInformation("Starting batch insertion of {Count} records...", customers.Count);
            await context.BulkInsertAsync(customers, options =>
                {
                    options.UseAudit = true;
                    options.AuditEntries = auditEntries;
                    options.AuditMode = AuditModeType.ExcludeAll;
                    options.PostConfiguration = bulkOperation =>
                    {
                        bulkOperation.ColumnMappings
                            .Where(x => x.SourceName is nameof(Customer.CustomerID))
                            .ToList().ForEach(p => { p.AuditMode = ColumnMappingAuditModeType.Include; });
                    };

                    options.BatchSize = 100;
                    options.AutoMapOutputDirection = false;
                    options.InsertIfNotExists = true;
                    options.ColumnPrimaryKeyExpression = p => new
                    {
                        p.Name,
                        p.IsDeleted // This field causes an error
                    };
                },
                token);

            stopwatch.Stop();
            Logger.LogInformation("Batch insertion completed, time consumed {ElapsedMs} ms",
                stopwatch.ElapsedMilliseconds);
            
            await tcs.CancelAsync();

            // 获取插入的ID
            var insertedIds = auditEntries.Where(p => p.Action == AuditActionType.Insert).SelectMany(p => p.Values)
                .Where(p => p.ColumnName == nameof(Customer.CustomerID) && p.NewValue != null)
                .Select(p => p.NewValue.ToString()).ToList();
            Logger.LogInformation("Inserted IDs: {Ids}", string.Join(", ", insertedIds));
        }
    }

    private static List<Customer> GenerateCustomers(int count)
    {
        var list = new List<Customer>();

        for (var i = 0; i < count; i++)
        {
            list.Add(new Customer(Guid.NewGuid())
            {
                CustomerID = i,
                Name = "Customer_" + i
            });
        }

        return list;
    }

    public class EntityContext : DbContext
    {
        public EntityContext()
        {
        }

        protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
        {
            optionsBuilder.UseNpgsql(
                new NpgsqlConnection(
                    "Host=localhost;Port=5432;Database=PgBulkInsert;User ID=postgres;Password=myPassword;"));
            // or use SQL Server
            // optionsBuilder.UseSqlServer("Server=localhost,1434;Database=PgBulkInsert;User Id=sa;password=myPassw0rd;TrustServerCertificate=True");
            optionsBuilder.UseLoggerFactory(LoggerFactory);
            optionsBuilder.EnableSensitiveDataLogging().EnableDetailedErrors();
            base.OnConfiguring(optionsBuilder);
        }

        protected override void OnModelCreating(ModelBuilder modelBuilder)
        {
            base.OnModelCreating(modelBuilder);
            modelBuilder.Entity<Customer>(b =>
            {
                b.ConfigureByConvention();
            });
        }

        public DbSet<Customer> Customers { get; set; }
    }


    public class Customer : FullAuditedAggregateRoot<Guid>
    {
        public Customer(Guid id) : base(id)
        {
            CreationTime = DateTime.UtcNow;
        }

        public int CustomerID { get; set; }
        public string Name { get; set; }
    }
}

[JonathanMagnan]

Improved version:

Hello @LEIRONGHUA,

Thank you for reporting this issue. My developer successfully reproduced it last week.

We had a meeting this morning to look for a solution. This is definitely a normal case that is currently not handled by our library.

Since the column is not expected to be inserted, our library currently skips it, so therefore it cannot be used in the primary key at this moment.

Meanwhile, one alternative solution, if the value of IsDeleted is always equal to 0, is to use the InsertPrimaryKeyAndFormula option, such as:

options.InsertPrimaryKeyAndFormula = "IsDeleted = 0";
options.ColumnPrimaryKeyExpression = p => new
{
    p.Name
};

We are currently investigating how to make a better fix.

Best Regards,

Jon