Notebook Review forum archive – NBRCHIVE

[DLoa]

6 hours ago, Sandy Bridge said:

The great news is we shouldn't be dependent on the Archive Team for the forums, although if they were also successful, all the better.  Our motley crew of NBR archivers has now covered (with minor and still-shrinking exceptions) all of the forums except What Notebook Should I Buy, Toughbook Buy/Sell/Trade, Site Suggestions, NBR Marketplace, and Off-Topic.  That includes the Windows forum, and the thread you linked should be included there too (although I didn't archive that one personally, so I can only vouch for it not falling into one of our known exceptions).

It's definitely handy learning ways to make personal archives.  We used HTTrack for parts, but JSoup (a Java library) for the majority.  If you have time this week to dig into Open Wayback, you may be the first to validate that the Internet Archive archive is working.

HTTrack, a friend just mentioned and I'd have given a shot earlier last week - yet again it might be redunddant and using precious bandwidth for better organised and concerted efforts like yours. 

It is the 11h Hour, and it looks like NBR is down for maintenace - it might it.  I was trying to reply to @jc_denton , thanks for the reply if you read this post here.

A quick search at Open Wayback didn't bringu up anything.  Let us know when it is live, and I check it is working, no doubt.

Thank you for your help brindfing over to a new chapter "Sandy Bridge".

D

[Samir]

2 hours ago, RS4 said:

Can we inject the archive into this forum instead of keeping it separate? I doubt the old owners will care, they are shutting down the forums without care and not even willing to sell, it is like trash to them.

This would almost be an undertaking as big as the archive because the archive would have to be 'mined' for the post data and then that would have to be inserted into the database.  Technically it is possible, but there are a few caveats to doing something like that since usernames, etc would have to be created that don't exist here or may under a different name.

Still, this would be the best scenario for the existing posts, and username conflicts could be avoided by prefixing 'NBR-' to existing posts, which could be merged with an existing username here if the person is the same.  A lot of hand holding and massaging to get something like this done, but I think this is one of the communities that would be strong enough to pull it off.  I think in years to come this could be done, but I wouldn't think anything like this would be attempted for a few years and it's quite daunting.

[Samir]

1 hour ago, Aaron44126 said:

The Notebook Review forums appear to have gone down at around 10:07 AM (U.S. Eastern) / 15:07 (UTC) today.  There is currently a note stating that they are "offline for maintenance" but I think that it is safe to assume that they are not coming back...

I was working with @Sandy Bridge on grabbing exception threads up to the last minute.  In the end, we ended up with around five longer Alienware threads that got cut short.  That was the last thing that we had to process.  So close, another ten or fifteen minutes and we would have been done with that.

There's probably some other minor issues in the archive that haven't been discovered.  We know that there is an issue with duplicate thread names in the same subforum (only one will make it into the archive) but it doesn't look like any of those were terribly consequential.  The vast majority of content from the covered subforums should be present.  Some stuff might be able to be fixed up later using data from archive.org.

Anyway.  I have my web server extracting a number of 7-zip archives that were submitted to me.  That process is going to take a few hours.  I will see about having the bulk of the data that we have collected available in basic form sometime this evening.

[Edit] As of 10:30 (U.S. Eastern) / 15:30 (UTC), they are now directing to this site with a number of poor replacements for NBR! https://www.techtarget.com/technologyguide/

Tremendous effort and results!  Unreal that so much data was rescued.  I'm so thankful for that.  I got everything I wanted for my 8760w and m6800 and started looking for some of my older laptops like the d610 and c400 and got what I could.  I remember the day I found NBR I was telling myself that 'someday' I'd search for all my laptop model numbers since everything that is ever known about them would be here.  I'm glad all that data was rescued so when my someday comes, I will find the results.

What we have experienced calls to light a greater problem on the Internet of the loss of digital history.  It is a serious problem that did not exist prior to the era of only digital.  And everyone is on the delete bandwagon--your bank who limits the number of years they can 'find' transactions, your cell phone company that doesn't give you bills beyond a certain date, and more.  As records in forms other than digital become scarce, the fragility of the digital format will become more evident as more data is lost.  It is a crisis for human history, which will stunted to the end of the paper age as it is the only current, proven long term medium that stands the test of time for more than 10 years.  This should be a wake up call for all of us that have digital archives, personal and professional.

That being said, attempting to print out NBR would have been just as ludicrous as turning it off.

Edited January 31, 2022 by Samir
can't print the whole thing

[Aaron44126]

Ha, well it's going to take a while to move everything that I downloaded to the web server...  I wonder if it would be faster to archive it, move the archive, and then unpack it again...

[Blacky]

Good job guys!

[MagicMike]

Well done guys, history (we) will not forget how you saved the day!

[Azone]

Thanks a ton for all the effort you all put in! Sad to see NBR is actually gone now, but it's great that all that valuable information has been preserved.

I was trying to do a backup of the OT forum (since I had some fun on there 10+ years ago), but NBR went down in the middle of my download. I've managed to get like 270 MB of JSON files (about 297K posts) - no idea how big that sub-forum actually was.

 

 

[Aaron44126]

3 minutes ago, Azone said:

I've managed to get like 270 MB of JSON files (about 297K posts) - no idea how big that sub-forum actually was.

That's one that we did not grab but I can tell you that it had around 720,000 total posts (not counting the "Motorized vehicles" subforum, another 19,000 posts).  archive.org should have most of it.

[Azone]

42 minutes ago, Aaron44126 said:

That's one that we did not grab but I can tell you that it had around 720,000 total posts (not counting the "Motorized vehicles" subforum, another 19,000 posts).  archive.org should have most of it.

Gotcha, thanks. Yeah, I did see archive.org had a lot of it, but figured I'd give it a go in case they missed anything.

 

If anyone here is interested in the incomplete set of OT posts, let me know.

[Samir]

5 hours ago, Aaron44126 said:

Ha, well it's going to take a while to move everything that I downloaded to the web server...  I wonder if it would be faster to archive it, move the archive, and then unpack it again...

Robocopy is really good at handling large operations like that from ssd to ssd with the /mt option that you can set up to 128 ime.  It will work your drives though!

[Aaron44126]

18 minutes ago, Samir said:

Robocopy is really good at handling large operations like that from ssd to ssd with the /mt option that you can set up to 128 ime.  It will work your drives though!

It did end up lowering the ETA somewhat but it is going to take until tomorrow.  (This isn't exactly SSD-to-SSD.  The F: drive is a remote network share.)

[Samir]

16 minutes ago, Aaron44126 said:

It did end up lowering the ETA somewhat but it is going to take until tomorrow.  (This isn't exactly SSD-to-SSD.  The F: drive is a remote network share.)

Hmmm...what did you set the /mt to?  Or did you leave it to the default of 8?

The reason I ask is because I think I caused one of my HGST enterprise 2TB drives to die with /MT:128 on a nightly remote backup.  Usually there wasn't much data and the files were small in size so wasn't so much head trashing.  But when it did get to some larger files and because the remote backup pipe is relatively low bandwidth (1-2MB/s), the result is that a bunch of files could get queued up and the drive could end up in seek city on both ends.

Your idea of creating a large files, sending it over and decompressing is actually going to be the fastest method.  I haven't done that since the win9x days, but it was always faster when dealing with thousands of small files.

[Sandy Bridge]

Well, everything I archived has been uploaded to Aaron's server.  Including compressed copies as well as uncompressed, and some unnecessary duplication of JavaScript and CSS, it's over 100 GB.  Please be patient as there are a lot of things to be organized and matched up - and the pace of the past week is not sustainable over the long term!

It's been a fun time preserving the knowledge.  I knew my XenForo program could be extended to back up larger amounts of data, and had fortuitously taught it to traverse forums in mid January, but never planned to teach it new tricks at such a breakneck speed.  But I told some friends that I needed something to keep me busy on the 21st, and what do you know I learn about NBR going down on the 22nd, and all of a sudden I had something to keep me busy for the rest of January.

It's been a team effort as well.  With only my own local resources and motivation, a couple orders of magnitude less would have been preserved.  Aaron's thread in the Precision forums was key in organizing and building momentum, and seeing various people asking about archiving/preservation across the forums was also important from a motivation standpoint.  It wasn't always glamorous or fast-paced, queuing up backups of huge forums that took hours - I think my longest-running one was 30 hours.  Yet towards the end, with a significant but rapidly dwindling amount of small things left to grab, and running on a lack of sleep and fast-paced music, it was great fun.  We were so close to reaching our last known threads to grab - partial archives were made of those last five - but compared to what looked likely a week ago, I'm chuffed with the result.

[i_pk_pjers_i]

I've posted this elsewhere but I might as well post it here too:

I archived the entire Clevo section in JSON format minus the subsections.

Here is my dump of the Clevo section: https://www.mediafire.com/file/wobk5svd7xh1sy7/NotebookReview_Clevo_2022-25-25_13-38-48.zip/file

I can't wait to see the Clevo owners' lounge subsection once that's available in read-only format again since I wasn't able to archive that myself.

[Sandy Bridge]

6 minutes ago, i_pk_pjers_i said:

I've posted this elsewhere but I might as well post it here too:

I archived the entire Clevo section in JSON format minus the subsections.

Here is my dump of the Clevo section: https://www.mediafire.com/file/wobk5svd7xh1sy7/NotebookReview_Clevo_2022-25-25_13-38-48.zip/file

I can't wait to see the Clevo owners' lounge subsection once that's available in read-only format again since I wasn't able to archive that myself.

I downloaded it on the 28th after seeing your post at our old home, but didn't have time to dive too deeply into it with the focus on all the other forums as well.  But it's definitely handy, especially if we find gaps that it can fill in.  We don't know of any yet, but haven't been able to take too fine-toothed of a comb to what we have yet.

I'm curious what you used for creating it?  The tools we were using were focused on a human-consumable format, but I could see JSON being more directly consumable if the goal were having it as an input to another computer program or database, and sometimes that's what I'm looking for.

[i_pk_pjers_i]

1 minute ago, Sandy Bridge said:

I downloaded it on the 28th after seeing your post at our old home, but didn't have time to dive too deeply into it with the focus on all the other forums as well.  But it's definitely handy, especially if we find gaps that it can fill in.  We don't know of any yet, but haven't been able to take too fine-toothed of a comb to what we have yet.

I'm curious what you used for creating it?  The tools we were using were focused on a human-consumable format, but I could see JSON being more directly consumable if the goal were having it as an input to another computer program or database, and sometimes that's what I'm looking for.

One of my friends pointed me towards https://github.com/patobrien/xenforo-backup which I didn't find when searching for tools before, so I guess his google-fu was better than mine even though we're both programmers lol!

I had to modify it to change a CSS class name, and then put in the URLs, a real user agent, ignore robots.txt, etc. I wasn't able to get it to dump the sub-sections because the CSS classes were way different and I didn't have time to fix it unfortunately.

I also dumped the MSI section since I have owned MSI laptops as well, although it's like way smaller for some reason so I'm not sure if it really did fully dump the MSI section, it's uploading right now but the link when it's done is here: https://www.mediafire.com/file/wk4oqgks5ryhk4a/NotebookReview+MSI+2022-27-27_02-23-24.zip/file

[Sandy Bridge]

18 minutes ago, i_pk_pjers_i said:

One of my friends pointed me towards https://github.com/patobrien/xenforo-backup which I didn't find when searching for tools before, so I guess his google-fu was better than mine even though we're both programmers lol!

I had to modify it to change a CSS class name, and then put in the URLs, a real user agent, ignore robots.txt, etc. I wasn't able to get it to dump the sub-sections because the CSS classes were way different and I didn't have time to fix it unfortunately.

I also dumped the MSI section since I have owned MSI laptops as well, although it's like way smaller for some reason so I'm not sure if it really did fully dump the MSI section, it's uploading right now but the link when it's done is here: https://www.mediafire.com/file/wk4oqgks5ryhk4a/NotebookReview+MSI+2022-27-27_02-23-24.zip/file

Interesting; I shouldn't be surprised that someone else has worked on XenForo specific backups, but I hadn't heard of that project before.  His code looks much neater than mine, although my code looked much less ramshackle two weeks ago, before the recent fast-paced additions.

The MSI section should be less than half the size of Sager, in my archives it was roughly 33-40% of the size.  So it depends on what degree of "way" smaller you mean as to whether that sounds expected or not.

[Sandy Bridge]

Some early statistics on what I archived - uncompressed, and excluding common JavaScript and CSS files, it weighs in at 45.5 GB.  This includes extant images, including those that were hotlinked from other sites, as they were included in the archive.

I was not the only archiver, so the overall total will be higher than that.

7-Zipped, what I archived totals 25.7 GB, including somewhat duplicated JS/CSS.  Once more, the total will be higher once others' efforts are included.

Including downloads of other archival efforts (the Clevo JSON one above and some Internet Archive efforts), I am 201.6 GB poorer in disk space than a week ago, directly attributable to this archiving.  Although this includes having duplication across both compressed and uncompressed copies.  I don't plan to keep everything in both compressed and uncompressed copies forever, likely only keeping the data most relevant to my laptops as uncompressed.

[Ionising_Radiation]

Incredible effort to save a veritable treasure trove of content, guides, images, opinions and conversations, @Aaron44126, @Sandy Bridge (incidentally, what a wonderful microarchitecture), and @Reciever for setting up EFGXT. I wish I could've contributed, but I don't have the know-how. 

[Sandy Bridge]

I should also call out that if it hadn't been for Charles Jefferies, I never would have known NBR was going down until whenever I next came across it in a search result or thought to swing by, which could well have been months from now.  Pieces fell into place.  Charles messaged NBR members about the site going down, I saw that message, saw Aaron's progress, realized that a program I'd written for another forum could be modified and re-applied to the task, and was able to help speed things up as a result.

But without that message from Charles, maybe an equal amount of data would have been archived, but maybe not.

And without his messages, surely dozens and more likely hundreds of other members wouldn't have learned what was going on until months down the line.

I'd definitely understand if he wants to take some time off, and is exhausted by the events of the past few weeks.  But I'm hopeful that someday he will choose to re-appear at one or more of the successor forums.

[Reciever]

I appreciate the kind words, I really do. I hate the circumstances that culminated in our current events at the same time still in a state of shock that it both happened as well as we even have this platform now. 

I am an average bloke who makes an effort to understand, that is all that I can claim. 

@Aaron44126is the real star here who took point on archiving NotebookReview, I am not sure how the relationship between himself and @Sandy Bridgeculminated, perhaps (likely) organically, providing a (what seemed to be) homebrewed application to save all those thousands (millions, I believe) of posts. I only assisted to some degree.

Even with the forum that we have now, if @Hiewdidnt already have a cursory interest in communities and forums that reaches beyond a decade he wouldnt have been able to drum up a forum that is worth using and I would have nothing to offer and simply watch the clock until NBR was gone. 

All the mentioned names above have had much more of a tangible effect on the result that we have now. Its not perfect, and there is more work to be done, but all I did was facilitate to the best of my abilities, and will continue to do so, so long as I have the privilege to do so. 

I could never fill in the shoes that Charles left behind, wish he was around. Maybe that will change in the future, hopefully it does even for a little while.

[Sandy Bridge]

The word "organic" as Reciever mentioned fits it really well.  I don't know @Aaron44126outside of January at NBR/EFGXT, but posted in his progress thread with what I'd done and (probably?) some links to my in-progress work showing that it really worked.  My archiving application is indeed homebrewed, using a Java library that I'd encountered at a previous job and found to be pretty handy for web crawling, but beyond that being custom for XenForo.  Its initial job was archiving threads, though not whole forums, and fixing broken hotlinked images where possible (if you know of any good ImageShack archives, or archives of other defunct image-hosting sites, please message me).  But it became clear that Aaron had the most progress towards the shared goal so far, and it didn't take much longer to realize he had the server-side knowledge and hardware to make it happen in a way that my experience didn't lend itself to nearly as well.

The funny thing is that I'd been mostly inactive at NBR for so long, that I didn't recognize @Recievereither.  Maybe with the right post, with an old avatar, a lightbulb would go off, but from what I've read about his NBR timelines, it's equally possible that he became active about the same time that I became fairly inactive.  But it became clear fairly quickly that he'd built a good reputation over the years when I was only visiting once in a blue moon.

I think the fatigue from not enough sleep is finally starting to settle in - I stayed up all night last night re-running edge cases to get a more complete archive, and I'm surprised the few hours of sleep I caught after NBR went down lasted as long as they did.  Hopefully Reciever can catch up on sleep as well - I don't know how he managed to get anything done while moving.  I moved in December and didn't even get the Internet set up for a few days, let alone find a mouse built after 1992 in my boxes, or have time to dabble in any of my side projects.

[Samir]

2 hours ago, Sandy Bridge said:

I should also call out that if it hadn't been for Charles Jefferies, I never would have known NBR was going down until whenever I next came across it in a search result or thought to swing by, which could well have been months from now.  Pieces fell into place.  Charles messaged NBR members about the site going down, I saw that message, saw Aaron's progress, realized that a program I'd written for another forum could be modified and re-applied to the task, and was able to help speed things up as a result.

But without that message from Charles, maybe an equal amount of data would have been archived, but maybe not.

And without his messages, surely dozens and more likely hundreds of other members wouldn't have learned what was going on until months down the line.

I'd definitely understand if he wants to take some time off, and is exhausted by the events of the past few weeks.  But I'm hopeful that someday he will choose to re-appear at one or more of the successor forums.

Yes, big hats off to Charles for alerting the userbase via PM.  I wouldn't have known either otherwise and would have lost all the capability to max out my laptops. :(

I hope one day he can share his story of the last days from the inside.  It would be a fascinating read and give a lot of us answers to burning questions.

[Samir]

1 hour ago, Sandy Bridge said:

The word "organic" as Reciever mentioned fits it really well.  I don't know @Aaron44126outside of January at NBR/EFGXT, but posted in his progress thread with what I'd done and (probably?) some links to my in-progress work showing that it really worked.  My archiving application is indeed homebrewed, using a Java library that I'd encountered at a previous job and found to be pretty handy for web crawling, but beyond that being custom for XenForo.  Its initial job was archiving threads, though not whole forums, and fixing broken hotlinked images where possible (if you know of any good ImageShack archives, or archives of other defunct image-hosting sites, please message me).  But it became clear that Aaron had the most progress towards the shared goal so far, and it didn't take much longer to realize he had the server-side knowledge and hardware to make it happen in a way that my experience didn't lend itself to nearly as well.

The funny thing is that I'd been mostly inactive at NBR for so long, that I didn't recognize @Recievereither.  Maybe with the right post, with an old avatar, a lightbulb would go off, but from what I've read about his NBR timelines, it's equally possible that he became active about the same time that I became fairly inactive.  But it became clear fairly quickly that he'd built a good reputation over the years when I was only visiting once in a blue moon.

I think the fatigue from not enough sleep is finally starting to settle in - I stayed up all night last night re-running edge cases to get a more complete archive, and I'm surprised the few hours of sleep I caught after NBR went down lasted as long as they did.  Hopefully Reciever can catch up on sleep as well - I don't know how he managed to get anything done while moving.  I moved in December and didn't even get the Internet set up for a few days, let alone find a mouse built after 1992 in my boxes, or have time to dabble in any of my side projects.

Very cool to hear the story on how all this came together.  It's almost like there's a movie plot in here somewhere, or at least a cool documentary. 😉

Get some much needed rest.  We'll be here when you wake. :)

[Ionising_Radiation]

5 hours ago, Reciever said:

I could never fill in the shoes that Charles left behind, wish he was around. Maybe that will change in the future, hopefully it does even for a little while.

I do wish there was some way to reach out to the old mods; they had a thankless job and performed it with aplomb. Maybe Charles might be interested in continuing as a moderator here, as might @Custom90gt and a few others...