Hello folks, I'm wondering what you guys use to organize and index data dumps. Id like to make an easily searchable library of username:passwords.

thanks for any input! 😛 :angel:

++1

wanna know about this too

(November 11, 2022, 03:47 AM)silentalpha Wrote: Hello folks, I'm wondering what you guys use to organize and index data dumps. Id like to make an easily searchable library of username:passwords.

thanks for any input! 😛 :angel:

First you need some spare servers. I assume you have something old in your basement but if not,
1) Find any local "liquidators" that might get occasional servers or massive number of PC's or laptops (when a business does a Win11 upgrade or whatever)

Then you need to set them up
2) Wipe them and re-install Ubuntu 20.04 Server (bare essentials) + MongoDB + ssh server (sudo apt install <XYZ> is your friend).
2b) Load the /etc/mongodb.conf file and set the "bind address" to 0.0.0.0

Then you clean the data to only what you want. Let's say you've got the combolists (can also be done for the database leaks)
3) mongoimport --type csv -c comboCollection -d comboDB filenames.txt

4) Go into mongo and create an Index
% mongo
...
>> use comboDB
>> db.comboCollection.createIndex({id:1},{unique:false})
....wait for 5 min

>> db.comboCollection.find({id:"[email protected]"})

(November 11, 2022, 09:07 PM)blackboar Wrote:

(November 11, 2022, 03:47 AM)silentalpha Wrote: Hello folks, I'm wondering what you guys use to organize and index data dumps. Id like to make an easily searchable library of username:passwords.

thanks for any input! 😛 :angel:

First you need some spare servers. I assume you have something old in your basement but if not,
1) Find any local "liquidators" that might get occasional servers or massive number of PC's or laptops (when a business does a Win11 upgrade or whatever)

Then you need to set them up
2) Wipe them and re-install Ubuntu 20.04 Server (bare essentials) + MongoDB + ssh server (sudo apt install <XYZ> is your friend).
2b) Load the /etc/mongodb.conf file and set the "bind address" to 0.0.0.0

Then you clean the data to only what you want. Let's say you've got the combolists (can also be done for the database leaks)
3) mongoimport --type csv -c comboCollection -d comboDB filenames.txt

4) Go into mongo and create an Index
% mongo
...
>> use comboDB
>> db.comboCollection.createIndex({id:1},{unique:false})
....wait for 5 min

>> db.comboCollection.find({id:"[email protected]"})

Good advice. It all depends on the size of the data and how frequently one searches for stuff there in

[quote pid="805149" dateline="1668273497"]
Good advice. It all depends on the size of the data and how frequently one searches for stuff there in
[/quote]

Searching through a file (even on an SSD drive which adds $$$) is much slower than searching off an indexed field.

Assuming you're searching for you, your friends (or enemies), taking the 30-60 min to load and index it, quickly makes it worth the trouble. Searches come back in 1-3 seconds if the email field is indexed.

Using "grep" between your contact list and a bunch of leaks will often die (without error). So mongoDB (which doesn't care about the correct schema ahead of time) is the best way to go (I've found).

Database Management solutions

You definitely want to avoid getting bogged down into cleaning/loading small breaches (unless you know you or your target is in that breach).

The big ones should be managed in a DB in an indexed field (LinkedInScraped, WattPad, Zynga).

But take Cit0day. Basically it's a two field dumped combo list. If you had it loaded into mongoDB and had the email/username field indexed, you could search on someone and have all passwords returned. And probably you'd notice a pattern.

Same word, same number, same special character at the end.

And if you new their "password vector" and had several hashes for the same person, you could use that information to attack that hash using rules in hashcat.

If you tried to do that without an index database (grepping through the 6550 files) you'd be searching for hours each time. And people often have multiple emails. Or usernames.

With an index, you could be searching EVERYONE in your local PTA when they send out emails without using the BCC. And the search of everyone comes back in seconds with all their passwords they've ever used.

I still haven't seen a better solution.