C# Remove Duplicate Items (Using Delimitation)

[C#] Remove Duplicate Items (Using Delimitation) - Printable Version

+- Hack Forums (https://hackforums.net)
+-- Forum: Tech (https://hackforums.net/forumdisplay.php?fid=444)
+--- Forum: Coding (https://hackforums.net/forumdisplay.php?fid=151)
+---- Forum: Software Development (https://hackforums.net/forumdisplay.php?fid=118)
+---- Thread: [C#] Remove Duplicate Items (Using Delimitation) (/showthread.php?tid=6072531)

Remove Duplicate Items (Using Delimitation) - draxyee - Apr 28, 2020

If List1[0] = "Test|543543"
And List2[5] = "Test"

Then don't include it in FinalList.

Current code:

PHP Code 


<?php 
List1 = List1.Select(s => s.Split('|')[0]).Except(List2).ToList(); 

However, it only adds the delimited value to the list (Test), not the full value which I want (ex: Test|543543)

Can anyone help?

RE: Remove Duplicate Items (Using Delimitation) - Porn - Apr 28, 2020

Here's a little program to remove dupes in c#

Code 
       static void Main(string[] args)

        {

            var sr = new StreamReader(File.OpenRead(@"C:\Users\yourName\Desktop\dupes.txt")); //input

            var sw = new StreamWriter(File.OpenWrite(@"C:\Users\yourName\Desktop\clean.txt")); //output

            var lines = new HashSet<int>();

            while (!sr.EndOfStream)

            {

                string line = sr.ReadLine();

                int hc = line.GetHashCode();

                if (lines.Contains(hc))

                    continue;

                lines.Add(hc);

                sw.WriteLine(line);

            }

            sw.Flush();

            sw.Close();

            sr.Close();

        }

RE: Remove Duplicate Items (Using Delimitation) - draxyee - Apr 28, 2020

(Apr 28, 2020 02:46 PM)Porn Wrote: Here's a little program to remove dupes in c#

Code
static void Main(string[] args) { var sr = new StreamReader(File.OpenRead(@"C:\Users\yourName\Desktop\dupes.txt")); //input var sw = new StreamWriter(File.OpenWrite(@"C:\Users\yourName\Desktop\clean.txt")); //output var lines = new HashSet<int>(); while (!sr.EndOfStream) { string line = sr.ReadLine(); int hc = line.GetHashCode(); if (lines.Contains(hc)) continue; lines.Add(hc); sw.WriteLine(line); } sw.Flush(); sw.Close(); sr.Close(); }

That isn't efficient enough, but thanks.

We're handling hundreds of thousands of items.

If anyone knows about LINQ and could modify my code it would be really appreciated!

RE: Remove Duplicate Items (Using Delimitation) - itcorp - Apr 28, 2020

You can use distinct buddy https://docs.microsoft.com/en-us/dotnet/api/system.linq.enumerable.distinct?view=netcore-3.1

RE: Remove Duplicate Items (Using Delimitation) - BahNahNah - Apr 28, 2020

(Apr 28, 2020 04:20 PM)itcorp Wrote: You can use distinct buddy https://docs.microsoft.com/en-us/dotnet/api/system.linq.enumerable.distinct?view=netcore-3.1

Distinct removes duplicates from a single list.

This will work:

Code 
var final = list1.Where(x => !list2.Contains(x.Split('|').First()));

RE: Remove Duplicate Items (Using Delimitation) - Shadow_Cloud - Apr 28, 2020

(Apr 28, 2020 02:48 PM)draxyee Wrote: That isn't efficient enough, but thanks.

We're handling hundreds of thousands of items.

If anyone knows about LINQ and could modify my code it would be really appreciated!

This will becoming slower as your collections grow. Have you considered a different data structure for what you are trying to achieve?
If you swap from list to dictionary you'll need to add some exception handling but it should handle large quantities significantly better.

Without seeing the data sample it's hard to comment but it might be worth doing a distinct on these if you start out with duplicates. I imagine you might be spending a bit of time in the split as well. If I were doing this on a large scale I'd probably save these individually so I don't need to perform the split.

It might also be worth breaking this down into set size lists. You could have different threads process the different list, it would be more work for the CPU but it might actually be faster if you can do more in parallel.

If you can supply the data (even if we are just talking a comparable test data) and we all have the same base to work from and the same output to achieve it could be a fun little experiment while working through any edge cases.