[draxyee]

If List1[0] = "Test|543543"
And List2[5] = "Test"

Then don't include it in FinalList.

Current code:

PHP Code


<?php 
List1 = List1.Select(s => s.Split('|')[0]).Except(List2).ToList(); 


However, it only adds the delimited value to the list (Test), not the full value which I want (ex: Test|543543)

Can anyone help?

[Porn]

Here's a little program to remove dupes in c#

Code

       static void Main(string[] args)
       {
           var sr = new StreamReader(File.OpenRead(@"C:\Users\yourName\Desktop\dupes.txt")); //input
           var sw = new StreamWriter(File.OpenWrite(@"C:\Users\yourName\Desktop\clean.txt")); //output
           var lines = new HashSet<int>();
           while (!sr.EndOfStream)
           {
               string line = sr.ReadLine();
               int hc = line.GetHashCode();
               if (lines.Contains(hc))
                   continue;

               lines.Add(hc);
               sw.WriteLine(line);
           }
           sw.Flush();
           sw.Close();
           sr.Close();
       }

[draxyee]

Here's a little program to remove dupes in c#

Code

       static void Main(string[] args)
       {
           var sr = new StreamReader(File.OpenRead(@"C:\Users\yourName\Desktop\dupes.txt")); //input
           var sw = new StreamWriter(File.OpenWrite(@"C:\Users\yourName\Desktop\clean.txt")); //output
           var lines = new HashSet<int>();
           while (!sr.EndOfStream)
           {
               string line = sr.ReadLine();
               int hc = line.GetHashCode();
               if (lines.Contains(hc))
                   continue;

               lines.Add(hc);
               sw.WriteLine(line);
           }
           sw.Flush();
           sw.Close();
           sr.Close();
       }

That isn't efficient enough, but thanks.

We're handling hundreds of thousands of items.

If anyone knows about LINQ and could modify my code it would be really appreciated!

[itcorp]

You can use distinct buddy https://docs.microsoft.com/en-us/dotnet/...etcore-3.1

[BahNahNah]

You can use distinct buddy https://docs.microsoft.com/en-us/dotnet/...etcore-3.1

Distinct removes duplicates from a single list.

This will work:

Code

var final = list1.Where(x => !list2.Contains(x.Split('|').First()));

[Shadow_Cloud]

That isn't efficient enough, but thanks.

We're handling hundreds of thousands of items.

If anyone knows about LINQ and could modify my code it would be really appreciated!

This will becoming slower as your collections grow. Have you considered a different data structure for what you are trying to achieve?
If you swap from list to dictionary you'll need to add some exception handling but it should handle large quantities significantly better.

Without seeing the data sample it's hard to comment but it might be worth doing a distinct on these if you start out with duplicates. I imagine you might be spending a bit of time in the split as well. If I were doing this on a large scale I'd probably save these individually so I don't need to perform the split.

It might also be worth breaking this down into set size lists. You could have different threads process the different list, it would be more work for the CPU but it might actually be faster if you can do more in parallel.

If you can supply the data (even if we are just talking a comparable test data) and we all have the same base to work from and the same output to achieve it could be a fun little experiment while working through any edge cases.