UseNetGrab

I'm having a lot of trouble implementing this *great* hack on F2S's servers (freedom2surf). I'm getting 500 internal server errors, and yes, I uploaded in ascii, chmod the script and dir to 755, checked and double checked and checked again all the url's, username, password, and everything. I'm on server32. Also followed the author's instructions to the letter. I just think there might be something in it that f2s doesn't like...

Since this site is hosted by f2s also, I thought maybe you guys would have some idea of what's wrong. Well, have a look at this innovative script and let me know what you think.

If you want to visit the script's author's website, check out [stitt.org]

thanks in advance everyone! ;-)

See these:

use Date::Manip;
use News::NNTPClient;
use DBI;

I'm sure the problen is that f2s doen't have these modules installed... :(

I believe the script is designed to be run command line (executed with in telnet shell). If you try and execute it buy pulling it up in your browser you'll probably get a 500 error.


I am also having problems with usenetgrab. I run the script (commandline) and It goes through and grabs the headers and messages and sticks them in the database, but when it gets to the point in the script where its supposed to sort and thread the messages and stick them into the Phorum I get this error:

DBD::mysql::st fetchrow_array failed: fetch() without execute() at ./usenetgrab.sh line 219

Any suggestions.

OB

OB, are you using f2s also?
if so, where did you run the script at command line?

I'm not sure about your DBD::... error off the top of my head, but I'm gonna look into it and post another messsage later today.

I am not hosting on f2s.

(however, there is a cgi script that allows you to run some commands and command line scripts from your web browser. Goto www.cgiextremes.com and search for commander. Not sure if it works on every server, but I have had good luck on accounts that only allow FTP access. It's a major security risk, so I recommend that if you do use it, don't make it publically accessible).

Now back to my saga...

I have attempted to run the UsenetGrab script on 4 separate machines. All running RedHat (5.2 or greater). I thought the problem might be that the script was designed to run using an older DBI module, so on my personal linux box I installed and uninstalled every older and newer version of Perl DBI that I could find. Most attempts to run the script gave me that same DBI Fetch-error, the really old modules gave me compilation errors.

I have gone into the script and made some minor changes in an effort to get UNG to work. I really don't think I have made any progress in that arena, as my Perl kung fu isn't as strong as I'd like it to be.

Thanks,

OB

OB, sorry for the delay. This looks like a mangled "No database selected" or "invalid database selected" error message.

i think the syntax you want for:

$dbh = DBI->connect("DBI:mysql:database=mysql;host=localhost", "dba", "",
{
RaiseError => 1
}
) || die "Database connection not made: $DBI::errstr";

should be:

$dbh = DBI->connect("DBI:mysql:yourdbname:localhost", "dba", "", ... ) or die;


just a suggestion, hope it helps
mojo

Sorry, no luck.

The script does connect to the database just fine, as it completes the process of pulling the newsgroups articles and headers and stuffing them into the database tables. Where it chokes is when it supposed to takes those articles and headers, sort them and stick them into phorum tables as threaded articles.

I am pretty confident that the problem is with the syntax for the script's "do while" loops and it's fetch_array syntax. I think it was probably written for an older perl or DBI version, although I am not sure.

Thanks for the help.

OB

I too am having problems running usenetgrab. I tired about two months ago and gave up after mailing the author who suggested a couple of things, but it still didn't work.

Last week I noticed he'd put out a new version so gave that a try. Now when I run it I get the following error:

Inserting Messages that do not refer to another into test
Inserting Messages without references to previous messages into test
Inserting 3419,
DBD::mysql::db do failed: Column 'email' cannot be null at ./usenetgrab-latest2.pl line 492.
SQL query: INSERT INTO test
(id, author, email, datestamp, subject, host, thread, parent, email_reply,Approved)
values
('7', 'dave', NULL, NULL, NULL, NULL, '7', '0','0','Y') failed with error: Column 'email' cannot be
null at ./usenetgrab-latest2.pl line 492.

But if I try making the email field accept a null value, it goes on to complain about nulls in all fields, not just email.

I got back to the author on this, but he's not got back to me. Guess he's too musy to mess about with losers like me, but I'd love to know why it's working on his setup an not mine. I'm running RedHat 6.2 with Phorum 3, NNTPClient-0.36 and DBI-1.14, and mysql-3.23.34.

JJ

Hello Jonathan

I have just seen your post about UseNetGrab (and will e-mail you this answer too, in case you have not clicked the "Email replies to this thread, to the address above." button)

The problem that you describe is exactly what I am having.

Did you manage to resolve it and get UseNetGrab to work?

I have noted that the FO_* tables appear to be populated correctly.

However, the script is still failing at line 505 with:

BD::mysql::db do failed: Column 'datestamp' cannot be null at
/home/jon/bin/usenetgrab.pl line 505.

SQL query: INSERT INTO ng_altgamesmicrosoft
(id, author, email, datestamp, subject, host, thread, parent,
email_reply,Approved)
values
('2', 'noSPAM', 'none', NULL, NULL, NULL, '2', '0','0','Y')

failed with error: Column 'datestamp' cannot be null at /home/jon/bin/usenetgrab.pl line 505.

For some reason, the variables $datestamp, $subject and $host are empty. I really feel I should be able to find out myself why, but I am not very well acquainted with Perl. :-((

I have a really smart application and a real need to get this working. I think the idea is great... Just the last few lines of the code seem not to work. :-((

If you or anyone else can shed some light on the situation, I would be really VERY happy.

TIA

S

> failed with error: Column 'datestamp' cannot be null at /home/jon/bin/usenetgrab.pl line 505.

That's as far as I got as well. When I asked the author about it he suggested making the 'datestamp' field able to be null, but when I did that I found the same error came up for EVERY field in that table (I have to take NOTNULL off them all!), so that's obviously not the problem. Something isn't passing the values into the table, but I think all that's hidden away in the modules somewhere.

He's not got back to me since then.

I too would be really happy if we could make this work, but like many scripts like this, it only seems to work for the author! Time to learn perl. Let's meet up again in five years time... ;-)

JJ

yep.

Similar problem.
Solved line 477

orignal code :
$sqlstmt="SELECT PosterName, PosterEmailAddy, Subject, PostedDate, PostingHost,ArticleId FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($author,$email,$subject,$datestamp,$host,$UsenetId) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;

replace with :
$sqlstmt="SELECT PosterName FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($author) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;
print "author="."$author"."\n";

$sqlstmt="SELECT PosterEmailAddy FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($email) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;
print "email="."$email"."\n";

$sqlstmt="SELECT Subject FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($subject) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;
print "subject="."$subject"."\n";

$sqlstmt="SELECT PostedDate FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($datestamp) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;
print "datestamp="."$datestamp"."\n";

$sqlstmt="SELECT PostingHost FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($host) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;
print "host="."$host"."\n";

$sqlstmt="SELECT ArticleId FROM FO_UsenetArticles WHERE Id=$ArticleId";
my ($UsenetId) = $dbh -> selectrow_array($sqlstmt)
|| die "SQL query: $sqlstmt failed with error: ".$dbh->errstr;

works fine here

Any clues about posting back to NG?
Couldn't implement the newsportal function.

Mitia

Thanks for the reply!

I've not managed to get your fix to work though. Now I get a different error. It seems to download the articles OK, then puts the first one into the thread, only to stop with an unknown error:

Reading 0 messages from alt.railway.locomotives.uk.class37.....
Now reading messages from to 2318
Reading Article 2318
NNTPERROR: 501 header not in index
NNTPERROR: 501 header not in index
'<LyEv8.53134$C21.9490812@news6-win.server.ntlworld.com>'johnson58@ntlworld.com -100s
Inserting Messages that do not refer to another into test
Inserting Messages without references to previous messages into test
author=Ian Johnson
email=johnson58@ntlworld.com
subject=Spanish 37 Pictures
datestamp=2002-04-18 19:48:20
host=213.104.37.52
Inserting 2, Spanish 37 Pictures
SQL query: SELECT ForumLocationId,Thread from FO_UsenetXRef WHERE ForumNameId=1 AND ArticleId='(none)' failed with error: at ./usenetgrab-latest.pl line 348.

Any further clues on that one?

JJ