Edision EPG for FTA channels

[FTA4PA]

No it wasn't. I did add a bunch of customization to it so it isn't so hard coded. But, no buddy shouted out to me about wanting it. I think i would release it on a one by one basis for those that wanted to try and set it up.

Sounds good! I may take you up on that when we head into fall.

 

[Doug_M]

Sorry guys, I got really busy with work and haven't had a chance to write anything up yet. I'm a software developer, and my client is a state Department of Education, so things are crazy right now with school about to start back. I'll try to make it more of a priority in the next few weeks.

I can provide you this summary...

Looking forward to your writeup when you get the time. This sounds right up my alley.

 

[ancient]

This is a very interesting discussion to me, for a couple of reasons. I've been using zap2xml for quite some time now, but for one thing the original page that hosted it went away and apparently it's now only available on the Wayback Machine, which means the author currently has no way to update it if anything changes. But also, the one thing I have always liked about TitanTV is that you can add channels to your favorites from any market on the same account. But I have not found anything that will get listings from TitanTV and turn them into an XML file. iBoston appears to be doing it but after looking at his other thread I don't think the method he's using would work on a headless Linux server with no desktop (therefore no way to run Firefox), also I don't have anything that runs Windows at all.

If I were at all conversant in the Python computer language, which unfortunately I'm not, I understand they have a module called "Beautiful Soup" which is just made to do this sort of thing, but I could not make heads nor tails of it. Part of it is that I really don't understand how you automate logging into a page. I have to hand it to the author of zap2xml, he must be a genius programmer! Unfortunately his program is in Perl and I know even less about that than Python.

 

[iBoston]

This is a very interesting discussion to me, for a couple of reasons. I've been using zap2xml for quite some time now, but for one thing the original page that hosted it went away and apparently it's now only available on the Wayback Machine, which means the author currently has no way to update it if anything changes. But also, the one thing I have always liked about TitanTV is that you can add channels to your favorites from any market on the same account. But I have not found anything that will get listings from TitanTV and turn them into an XML file. iBoston appears to be doing it but after looking at his other thread I don't think the method he's using would work on a headless Linux server with no desktop (therefore no way to run Firefox), also I don't have anything that runs Windows at all.

If I were at all conversant in the Python computer language, which unfortunately I'm not, I understand they have a module called "Beautiful Soup" which is just made to do this sort of thing, but I could not make heads nor tails of it. Part of it is that I really don't understand how you automate logging into a page. I have to hand it to the author of zap2xml, he must be a genius programmer! Unfortunately his program is in Perl and I know even less about that than Python.

Its not too hard to parse TitanTV. They have a special ID representing each channel. You don't need firefox. You could use curl linux scripts and you would have to copy your cookies so that your logging in and fetching your specific channel information. Curl supports cookies.

 

[ancient]

Its not too hard to parse TitanTV. They have a special ID representing each channel. You don't need firefox. You could use curl linux scripts and you would have to copy your cookies so that your logging in and fetching your specific channel information. Curl supports cookies.

Yeah, that's kind of the problem - in an automated script that runs off a cron job, and assuming those cookies have a finite shelf life, how do you obtain fresh cookies at every run? If you have to manually fire up a browser and copy them over that means it will always be a manual process. I'm assuming there has to be some way to automatically login and obtain fresh cookies, after all zap2xml is doing a login for the services it supports, but since I am not a programmer so I hae no idea how something like that is done. This is one of those cases where (for those who do get programming) sample code would really be helpful, particularly if it is well-commented and/or not platform specific. I don't know if that "Beautiful Soup" Python module has any kind of support for logging in or if it's just for scraping or what. Personally if I have to attempt to do anything like that I'm much more comfortable with a Bash script but that's kind of Linux/MacOS specific, also you can't do a lot of things in a Bash script that can be done in one of those higher level languages (that I don't know). And even my attempts at Bash scripts have been limited to just a few lines that mostly just run other programs in a sequence.

 

[iBoston]

Update to this. They have updated the EPGimporter Plugin for Enigma2 receivers and added the very features that previously i had to modify/customize to make work. So now, you can use the off the shelf plugin and make the EPG work.

This is actually very nice, because i had only customized a .deb package, and couldn't get it working on the cheaper enigma2 receivers i had that utilized .ipk packages. I had some free time, and spent the last few days upgrading the images on my cheaper enigma2 receivers and discovered the features they had added.

With the updated plugin, i was easily able to add EPG to those receivers without any program modifications. It is really nice to now have EPG data on all the receivers in the house.

So, to recap, making EPG work is a 1.2.3 operation.

1. The now widely available - Enigma2 EpgImport Plugin.
2. Your channel specification file. Basically a mapping telling which channel is assigned to which enigma2 service.
3. URL link to the XMLTV format file that has all the EPG DATA.

I wrote a program that creates the XMLTV format file and it runs on my web server. So, technically, anyone could do 1 and 2, and then add my web server URL link to grab the EPGdata. #2 is only required for those that use custom service/channels. - But, if everyone was working from the Same Enigma2 channel list, (example, i know there are a couple US sellers selling enigma2 STBs with American Channels already programmed in) which means they would all be working from the same data. So, in that circumstance, #2 could be preformatted and deployed with the image, and of-course, the same could be true of #1, as they could pre-install the plugin. So, what i am saying is that EPG for U.S. for less technically inclined would be fairly easy.

Now the bad news. I scrape web sites to create my XMLTV EPG data, and I think would be technically illegal and immoral to share that data. - If there is a open source, or non commercial entity where you can obtain EPG information on channels, i could easily write a program to fetch that data and convert it to XMLTV format to be utilized. - So, if anyone is aware of such a source, I'd be interested in hearing about it.

 

[iBoston]

There are a couple channels where i could share the EPG data, as it is fixed weekly programming and wrote a separate feature into my program which allows me to manually create EPG data for those type of channels. So, that was my work and i can freely share it. So, if someone that is technically inclined would like to do a test....

97w KU - Smart LifeStyle TV
87w KU - Patient Channel
125w KU - PBS Feed channel

I will also add, that if anyone has EPG data on FEED channels where it is fixed weekly programming, they could give me that information, and i could add it to my program....

 

[FTA4PA]

Update to this. They have updated the EPGimporter Plugin for Enigma2 receivers and added the very features that previously i had to modify/customize to make work. So now, you can use the off the shelf plugin and make the EPG work.

This is actually very nice, because i had only customized a .deb package, and couldn't get it working on the cheaper enigma2 receivers i had that utilized .ipk packages. I had some free time, and spent the last few days upgrading the images on my cheaper enigma2 receivers and discovered the features they had added.

With the updated plugin, i was easily able to add EPG to those receivers without any program modifications. It is really nice to now have EPG data on all the receivers in the house.

So, to recap, making EPG work is a 1.2.3 operation.

1. The now widely available - Enigma2 EpgImport Plugin.
2. Your channel specification file. Basically a mapping telling which channel is assigned to which enigma2 service.
3. URL link to the XMLTV format file that has all the EPG DATA.

I wrote a program that creates the XMLTV format file and it runs on my web server. So, technically, anyone could do 1 and 2, and then add my web server URL link to grab the EPGdata. #2 is only required for those that use custom service/channels. - But, if everyone was working from the Same Enigma2 channel list, (example, i know there are a couple US sellers selling enigma2 STBs with American Channels already programmed in) which means they would all be working from the same data. So, in that circumstance, #2 could be preformatted and deployed with the image, and of-course, the same could be true of #1, as they could pre-install the plugin. So, what i am saying is that EPG for U.S. for less technically inclined would be fairly easy.

Now the bad news. I scrape web sites to create my XMLTV EPG data, and I think would be technically illegal and immoral to share that data. - If there is a open source, or non commercial entity where you can obtain EPG information on channels, i could easily write a program to fetch that data and convert it to XMLTV format to be utilized. - So, if anyone is aware of such a source, I'd be interested in hearing about it.

Hello iBoston! I am wondering if I could get this working on our Edision. As far as images I currently have TNAP 2.0, TNAP 4.1, OpenATV 6.4 and PurE2 6.3 with the latest EPG Importer plugin downloaded under each. I have a computer I keep on 24/365 to log data from my weather station so that could be used as a server for the xmltv guide data. I also have PuTTY and Filezilla installed on that system if needed although I am not well versed in using them. How would one go about setting this up? Do you have a sample channel specifications file from step 2 to get an idea how to build one for our channels? I known you scrape to get your data but if a person has a subscription to a service like Schedules Direct, who I believe supplies both xml and json, would that be a directly compatible format for the import? If not, I am not above trying to edit it to make it so. Thanks!

P.S. I know the Schedules Direct data is for hobbyist (non-commercial) use so I would not share it.

 

[iBoston]

Here would be your custom.sources.xml file : This tells epgimporter where to get the channel information from and where to get the xmltv format file you created.

<?xml version="1.0" encoding="UTF-8"?> <sources> <sourcecat sourcecatname="Custom Sources"> <source type="gen_xmltv" channels="/etc/epgimport/custom.channels.xml"> <description>192.168.1.52/xmltvepg.xml</description> <url>http://192.168.1.52/xmltvepg.xml</url> </source> </sourcecat> </sources>

The URL being where your getting the compatible enigma2 addon (epgimporter) file. I wrote a program to create this file. The program also fetches the SOURCE data and converts it to epgimport xmltv format.

So, you could write a script/program to fetch the 'Schedules Direct data' and convert it to the epgimport xmltv format.

Here is a sample custom.channels.xml : This file shows you the mapping between epgimport data and enigma2 service list

the channel id is the same ID as used in the sample xmltv file: The long id with the ' : ' colons is the enigma2 channel (service) ID. Thats how it knows how to link the two.

 <channel id="12316">1:0:1:6:50F:0:82A2E23:0:0:0</channel>   <!--  LPB  -->
 <channel id="12318">1:0:1:4:50F:0:82A2E23:0:0:0</channel>   <!--  LPB2  -->
 <channel id="12317">1:0:1:5:50F:0:82A2E23:0:0:0</channel>   <!--  LPB Create  -->

and here is the xmlt format : The <programme> header is the repeating xml portion where there is an entry for every program show and for every channel.... Ie, thousands of entries.... The more entries you have the MORE it bogs down your system when updates are made. This is why i chose to import only a few hours ahead. I then update the xmltv data every couple hours to keep the epg where i always have a 3 hour ahead epg.. I don't see any issue doing 12 hours or 24 hours ahead etc, but for my personal viewing habbits, it was a waste of data for me, as I don't care enough about tv to see whats on days from now, let alone more then really whats up NEXT!

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE tv SYSTEM "xmltv.dtd">
<tv generator-info-name="xmltv.co.uk" source-info-name="xmltv.co.uk">

  <programme start="20211010113000 -0000" stop="20211010120000 -0000" channel="12316">
    <title lang="en">MotorWeek</title>
    <desc lang="en">(New Science 10/09/2021 TVG).....2021 Acura TLX Type S ADAS calibration 1987 Lotus Esprit Turbo HCI 2022 Toyota GR 86 and 2022 Hyundai Santa Cruz 2021 MercedesBenz GLS 450.</desc>
  </programme>

</tv>

I have one ongoing issue i haven't been able to track down. The enigma2 epgimporter crashes every once in awhile when importing the data. I have yet to figure out what is causing the issue, and would be curious to see if you run into the same thing. I can reboot and re import the same file and it will work. So, im not sure where the issue lies. But, its doesn't happen too often and its a quick reboot and re-import options within the STB so its not that big of a deal.

 

[FTA4PA]

Thanks very much for the info iBoston! Seems pretty straight forward so as long as I can get the xmltv data in a readable format I'm hoping I can figure this out and get it going for us. Will let you know.

A question. I used Filezilla to take a look at the directory structure on the Edision. I'm guessing that both the custom.sources.xml and custom.channels.xml files are to be placed in the etc/epgimport folder as that directory is mentioned in your file. Is that correct or do they go somewhere else?

 

[iBoston]

Thanks very much for the info iBoston! Seems pretty straight forward so as long as I can get the xmltv data in a readable format I'm hoping I can figure this out and get it going for us. Will let you know.

A question. I used Filezilla to take a look at the directory structure on the Edision. I'm guessing that both the custom.sources.xml and custom.channels.xml files are to be placed in the etc/epgimport folder as that directory is mentioned in your file. Is that correct or do they go somewhere else?

That is correct, custom sources and custom channels go in the etc/epgimport folder. The xml epg file goes to any location accessible via http protocol. I use a personal internal webserver built on ubuntu, although, it can also be a public source.

 

[FTA4PA]

That is correct, custom sources and custom channels go in the etc/epgimport folder. The xml epg file goes to any location accessible via http protocol. I use a personal internal webserver built on ubuntu, although, it can also be a public source.

Thanks for confirming. I also have a computer running Ubuntu 24/7/365 so I should be good to go if the SD data is in a usable format or can be converted to one.

 

[iBoston]

Thanks for confirming. I also have a computer running Ubuntu 24/7/365 so I should be good to go if the SD data is in a usable format or can be converted to one.

Not sure what you mean, "If the SD data is in usable format"

 

[FTA4PA]

Not sure what you mean, "If the SD data is in usable format"

Schedules Direct, may have to convert it to something the plugin can use.

 

[iBoston]

Well, if they export in nice easy xml or json, then it should be fairly simple to write a conversion script.

 

[iBoston]

TitanTV guide for Arirang TV is not correct. It appears to be way off.. I ended up writing a script to decipher and load into Enigma2 from the following URL :

https://www.arirang.com/Tv/Tv_Index_schedule.asp

Now i can accurately see whats on Arirang.

 

[iBoston]

Well, I kinda wrote about this on another thread regarding 117w KU.

The BYU channel on 117w KU is BYUTV-International, which has a different schedule than any online schedule EPG i could find.... You can only see the BYU-I from BYU's own website. But, there's a catch.. BYU's website automatically detects your IP location and shows you the schedule American/International version that cannot be linked to manually.

I already have a 24/7 VPN Proxy service that i can access. I found a Firefox add-on called FoxyProxy. It allows me to specify URL patterns. So, on my computer that processes EPG, i have foxyproxy setup that when ever i access the BYU website, it does so through the VPN proxy which then places me in a location where i can then pull up the BYU-International schedule.

This is perfect, because most work needs to be done outside the VPN. Being it's URL specific, it requires no manual intervention. Thanks to this, my EPG program is now able to scrape the BYU-International schedule automatically..

 

[iBoston]

I successfully got an EPG for CGTN documentary. The start/stop times were in milliseconds, which took me a bit of time to figure that out.

Example :
startTime	"1654430400000"

I wrote a couple functions to convert milliseconds to date/time and back.
Function : ms2d = (milliseconds 2 date)
Function : d2ms = (date 2 milliseconds)

? ms2d("1654430400000")
6/5/2022 8:00:00 AM 

? d2ms(#6/5/2022 8:00:00 AM#)
1654430400000

After i got that figured out, the rest was easy.....

 

[wagonman76]

Is the FTA listings website down for everyone? It’s been down for me for like a month or two. Says account suspended.

 

[cyberham]

The site is gone. He used to tweet on twitter in parallel. On April 16, he tweeted six tweets as normal. And then silence. You would think he would have said something if his service was ending for financial or personal reasons? Maybe a health problem?