Notebook Review forum archive – NBRCHIVE

[Aaron44126]

5 hours ago, Blacky said:

This topic also needs to get archived

http://forum.notebookreview.com/forums/hardware-components-and-aftermarket-upgrades.27/

I downloaded this one myself.  It's a large subforum so it took too long to include in the first batch, but we have it.

[MagicMike]

Epic achievement guys!

Theoretically, if NBR were to be open for another week, would you be able to download all of it, inc. the lower reply count threads?

[Aaron44126]

5 minutes ago, MagicMike said:

Epic achievement guys!

Theoretically, if NBR were to be open for another week, would you be able to download all of it, inc. the lower reply count threads?

We actually have the lower reply count threads for most of the manufacturer forums now.  We are working through the last few.  Off topic, gaming, and marketplace forums are the main things that haven’t been grabbed.

If NBR goes down tomorrow, I’ll post an update with what we have and don’t have.  (I haven’t been making updates this weekend because things have been happening pretty quickly.)

[MagicMike]

Awesome stuff man! Is it possible to replicate those threads and posts here, as if they were posted here? Or would that be taking things too far?

[Aaron44126]

6 minutes ago, MagicMike said:

Awesome stuff man! Is it possible to replicate those threads and posts here, as if they were posted here? Or would that be taking things too far?

So I'm going to have the full thread list and it will look similar to what you saw at NBR.  See the archived Precision section for an example.  https://www.nbrchive.net/forum.notebookreview.com/forums/dell-latitude-vostro-and-precision.1051/

As for the threads themselves, they will basically look as you see now; I am planning to add pagination (maybe more like 50 posts per page instead of 10) and links back to the forum thread list.  I think that individual post permalinks are broken right now so I will fix that too.  But it won't be exactly like NBR.

These will be coming over the next few days/weeks.  Right now the focus is just on pulling down as much as possible and getting it staged online in some form as quickly as reasonable.

[slnotebook]

For anything you guys can't grab before NBR goes offline, you can probably get it from the Internet Archive that they posted a few days ago, right?

[Samir]

1 hour ago, Aaron44126 said:

So I'm going to have the full thread list and it will look similar to what you saw at NBR.  See the archived Precision section for an example.  https://www.nbrchive.net/forum.notebookreview.com/forums/dell-latitude-vostro-and-precision.1051/

As for the threads themselves, they will basically look as you see now; I am planning to add pagination (maybe more like 50 posts per page instead of 10) and links back to the forum thread list.  I think that individual post permalinks are broken right now so I will fix that too.  But it won't be exactly like NBR.

These will be coming over the next few days/weeks.  Right now the focus is just on pulling down as much as possible and getting it staged online in some form as quickly as reasonable.

Holy cow, this would be unreal.  But I think one of the issues with publishing this publicly is that it is 'too NBR' to be considered fair use, a derivative work, etc, and might get some legal attention because of it if the parent company chooses.

That be said, if this is behind closed doors for 6 months or so and then published, I don't think anyone would notice or care anymore even if they did.

[Aaron44126]

33 minutes ago, slnotebook said:

For anything you guys can't grab before NBR goes offline, you can probably get it from the Internet Archive that they posted a few days ago, right?

Right.  The archive.org guys have been crawling the site too so they may well have a copy of threads that we do not.  (They normally crawl around from time to time but I read that someone tipped them off to the closure so they're making a special effort to do a full crawl.)  In fact my thread index is going to link to archive.org for any threads that we do not have downloaded, so hopefully it will be "sort of" seamless to browse.

[Samir]

55 minutes ago, Aaron44126 said:

Right.  The archive.org guys have been crawling the site too so they may well have a copy of threads that we do not.  (They normally crawl around from time to time but I read that someone tipped them off to the closure so they're making a special effort to do a full crawl.)  In fact my thread index is going to link to archive.org for any threads that we do not have downloaded, so hopefully it will be "sort of" seamless to browse.

In one of the threads over at nbr, I saw that the crawl was complete and that they got 100% of it uploaded as warc files.  I copied the links so I'll paste it here too so everyone has it:  https://archive.org/download/technologyguide_forums_20220125

I haven't attempted to download any of it as I think a lot of people have been hammering it and it's at dialup speeds the last I saw someone post about it.  Once things cool down, it should be easy enough to get them.

Viewing these warc files though require a viewer to be installed, and my thinking was that if the viewer was installed here on this server with a copy of the archives, then all of the old site would be accessible directly here as well.

Edited January 31, 2022 by Samir
grammar

[Aaron44126]

12 minutes ago, Samir said:

In one of the threads over at nbr, I saw that the crawl was complete and that they got 100% of it uploaded as warc files.

Ok, this is news to me.  I’ll take a look and see what we have here.

[Samir]

17 minutes ago, Aaron44126 said:

Ok, this is news to me.  I’ll take a look and see what we have here.

Yeah, it's been hard to keep up with everything going on for sure.  And the person that posted this also was the one communicating with archive so he was able to call and verify this before posting.

I've even gone back to try to find the original post so I can link it here, but it was just a single post in one of the many threads about the site closing. 😞

I think I may have found it, but it seems to have been edited because there's nothing about contacting archive and confirming it's done:

http://forum.notebookreview.com/threads/all-technologyguide-forums-will-close-jan-31-2022.837524/page-56#post-11139925

Edited January 31, 2022 by Samir
looking for the original post...and found?

[Samir]

Okay, so while the post I found says it is done, I found the status page of the actual archive bot and if you put in 'notebook' into the field, you will find that it is still archiving the site with 50GB+ and over half complete, but almost half still to go:  http://dashboard.at.ninjawedding.org/3

Edited January 31, 2022 by Samir
more stats!

[Aaron44126]

28 minutes ago, Samir said:

Okay, so while the post I found says it is done, I found the status page of the actual archive bot and if you put in 'notebook' into the field, you will find that it is still archiving the site with 50GB+ and over half complete, but almost half still to go:  http://dashboard.at.ninjawedding.org/3

This looks like it is crawling the main site (www.notebookreview.com) and not the forum site.

The WARC files for the forum add up to a bit over 30 GB.  I know they're compressed so it's really more data than that (possibly significantly more).  I wonder if it is mostly just the thread contents (text) or if they have images and attachments in there too.  I see there is a desktop viewer so I will download them and dig in soon, just to see.  I am downloading them but at current speeds it is going to take forever as you said.  (Over three hours just for the first 5GB chunk.)

[Edit] Found the link that you were looking for.  I was following this thread, but missed it.  http://forum.notebookreview.com/threads/all-technologyguide-forums-will-close-jan-31-2022.837524/page-56#post-11139925

[Sandy Bridge]

That would be great if they have the main site.  I downloaded one of their WARC archives a few days ago and forgot about it; the Web Recorder desktop software is now indexing it.

The Latitude/Vostro/Precision section looks great, even the forum styling, with page navigation working (although a few gaps of course; sorting the columns takes you to the "real" NBR, although you might not realize it right away).  Though I agree with Samir that I probably would have started out with a "less" NBR version initially, just to allow some time to pass.

[Sandy Bridge]

Well, it's been over an hour, and the Web Recorder is still indexing it.  Stopped it and restarted, but it's still indexing.  I'll try moving it to my SSD, the HDD is getting hammered right now.

Web Recorder did open the unsuccessful 5.58 GB WARC on the 25th, so I know it works... hopefully the fact that it's taking forever means there's more stuff in this one.

[Samir]

2 hours ago, Aaron44126 said:

This looks like it is crawling the main site (www.notebookreview.com) and not the forum site.

The WARC files for the forum add up to a bit over 30 GB.  I know they're compressed so it's really more data than that (possibly significantly more).  I wonder if it is mostly just the thread contents (text) or if they have images and attachments in there too.  I see there is a desktop viewer so I will download them and dig in soon, just to see.  I am downloading them but at current speeds it is going to take forever as you said.  (Over three hours just for the first 5GB chunk.)

[Edit] Found the link that you were looking for.  I was following this thread, but missed it.  http://forum.notebookreview.com/threads/all-technologyguide-forums-will-close-jan-31-2022.837524/page-56#post-11139925

I believe you're right as the urls it's currently processing don't seem to be forum ones.  So that's great to hear. :)

I believe I saw a post where someone downloaded that first chunk and it was corrupted.  I hope it's not the source that is and this was just their download.

[Sandy Bridge]

7 minutes ago, Samir said:

I believe you're right as the urls it's currently processing don't seem to be forum ones.  So that's great to hear. :)

I believe I saw a post where someone downloaded that first chunk and it was corrupted.  I hope it's not the source that is and this was just their download.

That was me, but I was trying what they had posted on 1/25, probably on 1/26.  The new 5 GB file are from 1/26.

It still says it's indexing, but I am skeptical.  I wonder if it's the Webrecorder software not being able to handle 5 GB?  There's an advanced guide linked from https://guides.lib.vt.edu/webarchiving/openwarc (where I learned about Webrecorder), so if someone has the time to try that, we might find out the answer sooner.  I'm too busy filling in gaps in the archive to try it now.

[DLoa]

Hi,

This a great effort.  Hopefully the archiveteam.org will cover most of the forum(s).  It's be quite a ride, I learned python to use WKHTMLTOPDF and PDFKIT, as well as WGET.  I was about to start using wpull https://github.com/ArchiveTeam/wpull when I finally got my Py3 script with WKHTMLTOPDF running.

  I wished I found about  https://guides.lib.vt.edu/webarchiving/openwarc earlier,   just came across openwayback yesterday and it was Unix env only.   Let me kn ow if I can help later today.

My topic of chose were related Clevo P775TM1G Chassis.  I managed to save to PDF a dozen threads.  Are there threads on more general topics like this one in your archive?

http://forum.notebookreview.com/threads/windows-10-clean-installation-guide.781178/page-3

Thanks again for all the team work.

D

 

[Sandy Bridge]

41 minutes ago, DLoa said:

Hi,

This a great effort.  Hopefully the archiveteam.org will cover most of the forum(s).  It's be quite a ride, I learned python to use WKHTMLTOPDF and PDFKIT, as well as WGET.  I was about to start using wpull https://github.com/ArchiveTeam/wpull when I finally got my Py3 script with WKHTMLTOPDF running.

  I wished I found about  https://guides.lib.vt.edu/webarchiving/openwarc earlier,   just came across openwayback yesterday and it was Unix env only.   Let me kn ow if I can help later today.

My topic of chose were related Clevo P775TM1G Chassis.  I managed to save to PDF a dozen threads.  Are there threads on more general topics like this one in your archive?

http://forum.notebookreview.com/threads/windows-10-clean-installation-guide.781178/page-3

Thanks again for all the team work.

D

 

The great news is we shouldn't be dependent on the Archive Team for the forums, although if they were also successful, all the better.  Our motley crew of NBR archivers has now covered (with minor and still-shrinking exceptions) all of the forums except What Notebook Should I Buy, Toughbook Buy/Sell/Trade, Site Suggestions, NBR Marketplace, and Off-Topic.  That includes the Windows forum, and the thread you linked should be included there too (although I didn't archive that one personally, so I can only vouch for it not falling into one of our known exceptions).

It's definitely handy learning ways to make personal archives.  We used HTTrack for parts, but JSoup (a Java library) for the majority.  If you have time this week to dig into Open Wayback, you may be the first to validate that the Internet Archive archive is working.

[Blacky]

This is awesome!

[Custom90gt]

This is a huge effort and I totally appreciate it! 

[Aaron44126]

So on the day of the NBR close, I can confirm that we have pretty much everything that we were shooting for.  The exceptions as @Sandy Bridge mentioned are the "What notebook should I buy" forum, "Site suggestions and guidelines", "Off topic", and all of the marketplace forums.  We are working through some exception cases while there is still time but by and large we are in really good shape.

We were able to grab threads with rely counts below the thresholds previously posted.  There is no post threshold for any of the subforums that we have now.

What I need to do now is sift through all of the files and get everything prepped for hosting in a basic form.  "Quality of life" improvements will be coming later.

[RS4]

Can we inject the archive into this forum instead of keeping it separate? I doubt the old owners will care, they are shutting down the forums without care and not even willing to sell, it is like trash to them.

[Aaron44126]

Just now, RS4 said:

Can we inject the archive into this forum instead of keeping it separate? I doubt the old owners will care, they are shutting down the forums without care and not even willing to sell, it is like trash to them.

I'm not really "in charge" on that aspect, but I've been rather against this idea.  It just muddies things up too much, both technically and legally.  I don't think that we should put any risk on this forum because "eh probably TechTarget won't care if we copy a bunch of their stuff over" — especially as the community is at risk of fracturing as there are two other alternate forums that have been stood up.  I'd love it if users who have posted guides and such would copy those over to the new forum, though (or even one of the others).

[Aaron44126]

The Notebook Review forums appear to have gone down at around 10:07 AM (U.S. Eastern) / 15:07 (UTC) today.  There is currently a note stating that they are "offline for maintenance" but I think that it is safe to assume that they are not coming back...

I was working with @Sandy Bridge on grabbing exception threads up to the last minute.  In the end, we ended up with around five longer Alienware threads that got cut short.  That was the last thing that we had to process.  So close, another ten or fifteen minutes and we would have been done with that.

There's probably some other minor issues in the archive that haven't been discovered.  We know that there is an issue with duplicate thread names in the same subforum (only one will make it into the archive) but it doesn't look like any of those were terribly consequential.  The vast majority of content from the covered subforums should be present.  Some stuff might be able to be fixed up later using data from archive.org.

Anyway.  I have my web server extracting a number of 7-zip archives that were submitted to me.  That process is going to take a few hours.  I will see about having the bulk of the data that we have collected available in basic form sometime this evening.

[Edit] As of 10:30 (U.S. Eastern) / 15:30 (UTC), they are now directing to this site with a number of poor replacements for NBR! https://www.techtarget.com/technologyguide/