Daily Rotation Search and Hot Topics Progress

Shell City and/or Daily Rotation -- Got something to say?
Post Reply
bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Daily Rotation Search and Hot Topics Progress

Post by bob » 02/04/10, 9:53 pm

OK, this time I have it. I've concocted a script that removes duplicates from the database. Not implemented yet, but thoroughly tested. This one works -- solves the problem of flat out dupes, and dupes from the same story being posted multiple days. Also solves the problem of sites making slight changes to the title for the same story throughout the day... but that's a little bit iffier, depends on what they change and the nature of their links.

This should do wonders for the Hot Topics scripting too, since duplicates won't be counted as two hits anymore, but I need to work with that a little more before I know whether I can fully automate the process or not.

Should be fully implemented by Monday.
WYSIWTF

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 02/08/10, 4:12 pm

OK, think I have it. Tell me if you see any repetition. Some hincky points not visible to users... Thinking them through to clear them up.

Also, as predicted, Hot Topics is playing nicer -- see this:

http://www.dailyrotation.com/news/hottopics.php

Still needs work. I might go with a one word model instead of word pairs -- might be a lot of day-to-day repetition though. For example, microsoft, apple and google would place high in the scores daily... Once again, have to think it through.
WYSIWTF

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 02/18/10, 12:25 am

Hot Topics - not sure I can make it automatic at all. No matter how complicated and screwey a scheme I come up with for automatically discovering hot topice, several of the so-called topics are nonsense.

http://www.dailyrotation.com/news/hot.php

OK, still, maybe I'll go to hand picking them. I might not need to change them very often... I've set up a beta attempt on my test page, here, right under the top10 headlines. See what you think:

http://www.dailyrotation.com/index2.php

My idea is that the first row will be hand picked and change every day or so... Or at least new topics added and removed, kind of daily.... The second row is pretty permanent, since it contains topics of pretty much ongoing interest.

Or maybe I can just come up with a semi-permanent set of topics.

Any suggestions?
WYSIWTF

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 02/28/10, 9:36 pm

>>>>Or maybe I can just come up with a semi-permanent set of topics.


Our "All The Latest" feature's really getting a lot of use. Over the last couple hours alone, 60 hits. Any ideas for topics to add to it? Currently it has: Apple, Google, Microsoft, Linux, Windows, Security, Smartphones, Open Source, Games
WYSIWTF

Gerry
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 5727
Joined: 12/04/01, 12:00 am
Location: Perth, Western Australia
Contact:

Post by Gerry » 03/01/10, 2:13 am

Ubuntu

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 03/01/10, 5:36 am

I can do that --

There must be loads of more general terms that would result in turning up lots of headlines lots of people are interested in...?
WYSIWTF

Gerry
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 5727
Joined: 12/04/01, 12:00 am
Location: Perth, Western Australia
Contact:

Post by Gerry » 03/01/10, 5:44 am

currently the searches are returning nothing

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 03/01/10, 6:30 am

I'm working on it--date problem, since it's 3-1-10 here, Perl's having a hard time figuring day -1, -2, -3 etc.... "All the Latest" is supposed to go back 4 days, including today. Getting late here though, and I'm getting tired.... I'll set up something temporary and work on the problem tomorrow.
WYSIWTF

Gerry
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 5727
Joined: 12/04/01, 12:00 am
Location: Perth, Western Australia
Contact:

Post by Gerry » 03/01/10, 10:45 am

No problems mate. Not an issue for me, just letting you know.

If you wanted to stop issues like that it's pretty simple to set up a server on your own machine and then test the code locally before uploading it to the live server. Again just FYI in case it's what you want.

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 03/01/10, 10:22 pm

OK, got it fixed -- my original method was too simplistic, just subtracting from the current date....

This seems to work:


Code: Select all

my $timeYesterday = time - 24 * 60 * 60;
my ($year, $month, $day) = (localtime($timeYesterday))[5,4,3];
my $yd = sprintf ("%02d-%02d-%04d", $month+1, $day, $year+1900);


my $timeYesterday1 = time - 48 * 60 * 60;
my ($year, $month, $day) = (localtime($timeYesterday1))[5,4,3];
my $yd1 = sprintf ("%02d-%02d-%04d", $month+1, $day, $year+1900);


my $timeYesterday2 = time - 72 * 60 * 60;
my ($year, $month, $day) = (localtime($timeYesterday2))[5,4,3];
my $yd2 = sprintf ("%02d-%02d-%04d", $month+1, $day, $year+1900);

probably can be compressed a bit....

Thanks for the advice, Gerry. Actually I've got a notebook here I'm not using. Maybe I'll set it up to match my server, just for a test box.
Last edited by bob on 03/02/10, 12:09 am, edited 1 time in total.
WYSIWTF

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 03/01/10, 10:32 pm

BTW, Script above is thanks to

Thanks to http://luckylarry.co.uk/2009/07/getting ... localtime/
WYSIWTF

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 04/04/10, 12:55 am

Our "All the Latest" feature has gotten more than 7000 hits since we created it. I'd say that's worthwhile.

Any ideas for more terms to add to it? Currently it has: Apple, Google, Microsoft, Linux, Windows, Security, Smartphones, Open Source, Games, and Ubuntu. I think I might add "Censorship." Any others?
WYSIWTF

Gerry
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 5727
Joined: 12/04/01, 12:00 am
Location: Perth, Western Australia
Contact:

Post by Gerry » 04/04/10, 11:09 am

Oh nice idea! I'd click that one. :)
I answer rhetorical questions for my own enjoyment.

bob
BIG GIANT HEAD I Get Free Beer
BIG GIANT HEAD I Get Free Beer
Posts: 7565
Joined: 12/03/01, 12:00 am
Location: St. Louis
Contact:

Post by bob » 04/14/10, 1:14 am

Censorship's not working out. I need to rethink that one.... Taking it down. Click on it and you'll see why.... My search engine doesn't lend itself to boolian searches and the word "censor" doesn't come near to covering all possibilities.

http://www.dailyrotation.com/cgi-bin/DB ... rds=CENSOR
WYSIWTF

Post Reply