(March 27, 2022, 08:05 PM)HaxonGhost Wrote:

(March 27, 2022, 07:48 PM)hexen Wrote:

(March 27, 2022, 07:31 PM)Inj3ct0r Wrote: "a personal Have I Been Pwned type thing"

Yeah, well HIBP is okay but he doesn't put everything on it. I like having stuff locally. Just takes me forever to search through stuff.

---

(March 27, 2022, 07:05 PM)HaxonGhost Wrote:

(March 27, 2022, 06:58 PM)hexen Wrote: I keep this stuff in flat files and just use ripgrep to search through it. It works, it's flexible, but can be slow as shit when you have gbs or tbs of data to sort through. I was just thinking maybe there's a better way.

What do you guys do? Do you put it in a database? If so what sort of db do you use?

I'm just trying to get some ideas. I'm thinking of something like a personal Have I Been Pwned type thing.

Troy Hunt (HIBP) save it in Azure, I think he save only fraction of data as first name, last name, email id, username, not much tbh

I am also looking for some solution, thinking to use elasticsearch but it takes only 1 string to search anything
I PMed @pompompurin for same, waiting for his reply, he has better idea of how to do as he dumped intelx data which is almost same as HIBP but with txt fil

I was thinking of something like Elasticsearch, too. I gotta read up on it.

---

(March 27, 2022, 07:00 PM)Minori Wrote:

(March 27, 2022, 06:58 PM)hexen Wrote: I keep this stuff in flat files and just use ripgrep to search through it. It works, it's flexible, but can be slow as shit when you have gbs or tbs of data to sort through. I was just thinking maybe there's a better way.

What do you guys do? Do you put it in a database? If so what sort of db do you use?

I'm just trying to get some ideas. I'm thinking of something like a personal Have I Been Pwned type thing.

I personally keep all breaches in seperate folders and then use "cat file.txt | grep whateverIwant" to find the line I want.

Yeah, me too. I use ripgrep, it's a little faster, but still on a 100+gb file....takes a bit of time.

If you are gonna build a server for elasticsearch, u can use maximum of 64GB sticks, however its scalable
If possible, check with how to filter out multiple strings, if that works, elasticsearch is the best. You can grab 18TB Enterprise HDD from Canada which costs just $400 CAD & dump your entire data on it
I am still trying to figure out for this solution, investing $1000 USD for elastic server for personal use would worth time, discord used to use it however its using Apache Cassandra now

Thanks for the ideas. I might give this a try :)

(March 27, 2022, 08:15 PM)hexen Wrote:

(March 27, 2022, 08:05 PM)HaxonGhost Wrote:

(March 27, 2022, 07:48 PM)hexen Wrote:

(March 27, 2022, 07:31 PM)Inj3ct0r Wrote: "a personal Have I Been Pwned type thing"

Yeah, well HIBP is okay but he doesn't put everything on it. I like having stuff locally. Just takes me forever to search through stuff.

---

(March 27, 2022, 07:05 PM)HaxonGhost Wrote:

(March 27, 2022, 06:58 PM)hexen Wrote: I keep this stuff in flat files and just use ripgrep to search through it. It works, it's flexible, but can be slow as shit when you have gbs or tbs of data to sort through. I was just thinking maybe there's a better way.

What do you guys do? Do you put it in a database? If so what sort of db do you use?

I'm just trying to get some ideas. I'm thinking of something like a personal Have I Been Pwned type thing.

Troy Hunt (HIBP) save it in Azure, I think he save only fraction of data as first name, last name, email id, username, not much tbh

I am also looking for some solution, thinking to use elasticsearch but it takes only 1 string to search anything
I PMed @pompompurin for same, waiting for his reply, he has better idea of how to do as he dumped intelx data which is almost same as HIBP but with txt fil

I was thinking of something like Elasticsearch, too. I gotta read up on it.

---

(March 27, 2022, 07:00 PM)Minori Wrote:

(March 27, 2022, 06:58 PM)hexen Wrote: I keep this stuff in flat files and just use ripgrep to search through it. It works, it's flexible, but can be slow as shit when you have gbs or tbs of data to sort through. I was just thinking maybe there's a better way.

What do you guys do? Do you put it in a database? If so what sort of db do you use?

I'm just trying to get some ideas. I'm thinking of something like a personal Have I Been Pwned type thing.

I personally keep all breaches in seperate folders and then use "cat file.txt | grep whateverIwant" to find the line I want.

Yeah, me too. I use ripgrep, it's a little faster, but still on a 100+gb file....takes a bit of time.

If you are gonna build a server for elasticsearch, u can use maximum of 64GB sticks, however its scalable
If possible, check with how to filter out multiple strings, if that works, elasticsearch is the best. You can grab 18TB Enterprise HDD from Canada which costs just $400 CAD & dump your entire data on it
I am still trying to figure out for this solution, investing $1000 USD for elastic server for personal use would worth time, discord used to use it however its using Apache Cassandra now

Thanks for the ideas. I might give this a try :)

You're really going to listen to the moron? Whatever, not even gonna waste my time with you, if you can't tell he's a moron you're just as retarded as him and you should both commit suicide.

(March 27, 2022, 08:16 PM)thekilob Wrote:

(March 27, 2022, 08:15 PM)hexen Wrote:

(March 27, 2022, 08:05 PM)HaxonGhost Wrote:

(March 27, 2022, 07:48 PM)hexen Wrote:

(March 27, 2022, 07:31 PM)Inj3ct0r Wrote: "a personal Have I Been Pwned type thing"

Yeah, well HIBP is okay but he doesn't put everything on it. I like having stuff locally. Just takes me forever to search through stuff.

---

(March 27, 2022, 07:05 PM)HaxonGhost Wrote: Troy Hunt (HIBP) save it in Azure, I think he save only fraction of data as first name, last name, email id, username, not much tbh

I am also looking for some solution, thinking to use elasticsearch but it takes only 1 string to search anything
I PMed @pompompurin for same, waiting for his reply, he has better idea of how to do as he dumped intelx data which is almost same as HIBP but with txt fil

I was thinking of something like Elasticsearch, too. I gotta read up on it.

---

(March 27, 2022, 07:00 PM)Minori Wrote: I personally keep all breaches in seperate folders and then use "cat file.txt | grep whateverIwant" to find the line I want.

Yeah, me too. I use ripgrep, it's a little faster, but still on a 100+gb file....takes a bit of time.

If you are gonna build a server for elasticsearch, u can use maximum of 64GB sticks, however its scalable
If possible, check with how to filter out multiple strings, if that works, elasticsearch is the best. You can grab 18TB Enterprise HDD from Canada which costs just $400 CAD & dump your entire data on it
I am still trying to figure out for this solution, investing $1000 USD for elastic server for personal use would worth time, discord used to use it however its using Apache Cassandra now

Thanks for the ideas. I might give this a try :)

You're really going to listen to the moron? Whatever, not even gonna waste my time with you, if you can't tell he's a moron you're just as retarded as him and you should both commit suicide.

Seems like it's worth try. I mean the idea, not the suicide part.

(March 27, 2022, 08:33 PM)hexen Wrote:

(March 27, 2022, 08:16 PM)thekilob Wrote:

(March 27, 2022, 08:15 PM)hexen Wrote:

(March 27, 2022, 08:05 PM)HaxonGhost Wrote:

(March 27, 2022, 07:48 PM)hexen Wrote: Yeah, well HIBP is okay but he doesn't put everything on it. I like having stuff locally. Just takes me forever to search through stuff.

---

I was thinking of something like Elasticsearch, too. I gotta read up on it.

---

Yeah, me too. I use ripgrep, it's a little faster, but still on a 100+gb file....takes a bit of time.

If you are gonna build a server for elasticsearch, u can use maximum of 64GB sticks, however its scalable
If possible, check with how to filter out multiple strings, if that works, elasticsearch is the best. You can grab 18TB Enterprise HDD from Canada which costs just $400 CAD & dump your entire data on it
I am still trying to figure out for this solution, investing $1000 USD for elastic server for personal use would worth time, discord used to use it however its using Apache Cassandra now

Thanks for the ideas. I might give this a try :)

You're really going to listen to the moron? Whatever, not even gonna waste my time with you, if you can't tell he's a moron you're just as retarded as him and you should both commit suicide.

Seems like it's worth try. I mean the idea, not the suicide part.

You're just so fucking stupid, it's unbelievable people like you are allowed to exist.

(March 27, 2022, 08:05 PM)HaxonGhost Wrote: If you are gonna build a server for elasticsearch, u can use maximum of 64GB sticks, however its scalable
If possible, check with how to filter out multiple strings, if that works, elasticsearch is the best. You can grab 18TB Enterprise HDD from Canada which costs just $400 CAD & dump your entire data on it
I am still trying to figure out for this solution, investing $1000 USD for elastic server for personal use would worth time, discord used to use it however its using Apache Cassandra now

I've been thinking about doing something like this too. I find peoples solutions for it pretty interesting.

This is my unsorted/unorganized folder of 600gb+ I've been parsing a few of them so it's gone down from 1tb to 600gb

I just sift through these, everything else should be in a database so you can wildcard in seconds

(March 27, 2022, 07:00 PM)Minori Wrote: I personally keep all breaches in seperate folders and then use "cat file.txt | grep whateverIwant" to find the line I want.

you don't need to cat into grep. just do grep whateveriwant file.txt. also make sure to add -F so you're using fixed strings instead of treating whatever you're looking for as a regex. (unless you're using a regex ofc)

(March 27, 2022, 10:15 PM)fridaynights Wrote: This is my unsorted/unorganized folder of 600gb+ I've been parsing a few of them so it's gone down from 1tb to 600gb

I just sift through these, everything else should be in a database so you can wildcard in seconds

The other stuff you put in MySQL/MariaDB or something?

(March 27, 2022, 10:54 PM)hexen Wrote:

(March 27, 2022, 10:15 PM)fridaynights Wrote: This is my unsorted/unorganized folder of 600gb+ I've been parsing a few of them so it's gone down from 1tb to 600gb

I just sift through these, everything else should be in a database so you can wildcard in seconds

The other stuff you put in MySQL/MariaDB or something?

I just got this new PC and have nothing setup, just transferred a roughly 2tb of data + excluding comb and collections

i've seen people setup mongodb locally, you can setup elastic search if you'd like it's "ideal", a few other options that can work after optimizing them as well

I am following though, it'd be nice to see what everyone does

https://github.com/stefankueng/grepWin