website i'm starting ... your data mine (not crypto block mining )

[j_scheibel]

I thought i’d mention it here. I’ve decided to put together a website ( yourdatamine.com ) that will let you do adhoc data mining. There is nothing to see there yet. though hopefully by the end of the weekend there will be.

i’ve long since written my own versions of gradient boosting, random forests, tsne and done a genetic algorithm implementation. I occurred to me yesterday I could quite easily expose them to the world and let people do their own predictions/data mining by merely dropping in a training csv and a test csv and letting the code do the work. I doubt it’ll ever replace a data scientist, but i figured the ability for a layman to get a quick fast result for a low cost is well worth it to a lot of people.

I say cost, but it’s gonna be free till i figure out the interest. i’m going to advertise on explainxkcd.com at first and see if anyone uses it. I figure those readers are in my target demographic (at least till the rest of the world catches up).

If I do start charging i’ll be sure FTC is a payment option :)

[Wellenreiter]

Interesting project.

I think I will try it soon.
Just need to think about, what I’d like to mine for ;)

[j_scheibel]

it should be working now.

edit okay… now it should really be working. my internet connection issues are fixed :)

and please excuse some of the poor phrasing/grammar issues. I really need to rewrite some of it but i was trying to get it done in a weekend.

[j_scheibel]

I made the https://www.yourdatamine.com/ site a lot better yesterday. you can now pick settings for data mining and optimize those settings from training data. you can even try out a couple samples on the optimization tab without having data :) … i still haven’t started advertising it. i want at least a few more samples. But, soon!

[Wellenreiter]

The site is very good now, especially the examples help to get the meaning of the calculations. :D

did you advertise your site on the FTC Telegram or Discord?

[AcidD]

very interesting. I work for a big data firm and this is really nice to see. you should put links to your github on it!

[j_scheibel]

@Wellenreiter I added more samples last night. some are more dubious than others :) (as is the nature of data mining) I added some graphics to … favorite.ico and a big one int how to page (got them from shutter stock and edited them up in gimp). I haven’t advertised anywhere yet, i will definitely look in to doing it those places. i did join the 2 biggest facebook groups just so i share a link there but i think i dont think i’m quite what people expect in those groups.

@AcidD I actually haven’t put my code in github at all. I do stock analysis as well for personal investment and of course work on kaggle contests from time to time. It all uses some version of this code. so pretty much everything i have I kinda want to keep to myself till i get some sort of payout from it. Then well anyone can have the code. Is that greedy? its been a hobby since the netflix contest years ago. I’d just like to see something monetary wise before i give it to the world.

I also have a TSNE implementation that is linear runtime, i might add to the website some day. As well as my genetic stuff that does all kinds of magical things. But i’m about to retool that to work on making new features instead of solving the big problem (i think it’ll do a better job of that)

There will probably be 1 more significant update to the site where i add 2 or 3 more samples and add the ability to have features that are categorical via an optional 2nd header row (which can either be split automatically for you via one-hot-encoding, or compiled down to averages solution value for that particular categorical value, your choice)

[j_scheibel]

I know this is an old topic, but I figured I’d finished with an epilog of how it went. Well, 1 + year on. I gave up on it =/ . I just couldn’t get any interest. Just the nature of data mining. big companies hire people. Little ones don’t know enough to seekout solutions like that.

That being said. I’ve moved on to some new things using my data mining skills :) I give you…

https://securitiesminer.com

I finished it up more or less today (2019-11-07). it too might go the way of the dodo if I don’t get any interest here is as well. but i’m actually using the predictions personally for investing so… probably it’ll stay for the foreseeable future.

I actually quit my day job to work that and a few other projects. but as they are off topic i’ll leave it at that

[Wellenreiter]

@j_scheibel I’m sorry to read, that you didn’t get no people attrached by your work.
I visited the website and it seems easy to use but still requires some knowledge interpret the data displayed.
That is a common situation, as the site is a tool and you always need to know how to use a tool.

Is it correct, that you have enabled the monitoring on October 21th?
The look of the graphs suggest that.

Also would it be a high effort to include Cryptocoins in the database? ;)

I found shares, but ‘FTC’ and ‘BTC’ in the search field didn’t get any results ;)

[j_scheibel]

@wellenreiter well, i’ve been using it off and on for like 1.5 years. (no ui around it previously, i would just query my database directly) its been a lot of trial and error . the volatility line for example was essentially what i used to use… till i realized it was an overfit (made money for months then dramatic losses.).

I previously would reset the data from time to time when i would implement new things. This last reset was when i started my much less aggressive fitting of the data. I actually added some more symbols and fixed the industry and sector listings a few days ago. normally it would warrant resetting the data but I’m trying to keep the history running so there is a graph of sorts. I guess it’s a work in progress. As there always seems to be a new thing to add, mining improvement or data to update.

Cryptocoins have 2 problems. they dont have much data (as in there are no financials) and they are strongly influenced by currently unidentifiable/unquantifiable mechanisms.

Normally I provide global metrics on each row of data. so, if i was analyzing microsoft, I not only include the latest quarterly data and the daily price… moving sector and industry averages for that day. but i also include that day’s: silver, oil, gold, 10 year us treasury bond prices, various loan rates …etc.etc. crypto would really only have those global pieces of information. which means other than price history and moving price averages all cryptos would look a lot a like.

Sure I can add, algorithm, network hash rate, maybe get hold of daily volume. but those aren’t really indicators for future success. those are more after the fact mechanisms. some of them may be useful in trend analysis but that’s not really what the site is doing. It’s trying to take fiscal indicators and extrapolate out where the stock will be. trend does play in to it, but i try to minimize it’s influence.

Now if i had something more… quantifiable, that correlated to future crypto value… well then maybe. half dates would defintely be 1 of those such things, but i think i need more. i use something like 200+ financials indicators for companies. i would hope i could have something like 10 or more at least, that maybe updated either weekly, monthly or quarterly. (the shorter term the better) but i dont know of any such indicators

[Wellenreiter]

I see the problems you have to include crypto coins. Most or all coins follow identical patterns, as nearly all coins have a <coin> <-> Bitcoin pair, but no direct pair to FIAT, so if Bitcoin moves they move the same way.
Additionally all in the crypo coin world most of the moves in exchange rates are based on speculations and ‘news’ (notice the hyphens ;) ) rather than real data.

It will take some time until there data statistics will be available, that can build a base for data mining and analysis

[j_scheibel]

@wellenreiter I’m all for doing it for fun. :) but i’d put far less stock in it than… uh stocks. :) maybe at the end of the year after i finish my current project (a website to build and share networks) but before i do my last planned project (a video game which will take probably a year or more to do).

[Wellenreiter]

It sounds, like you have a lot of ideas and not time enough to develop them ;)

Video game: any in game payments? in FTC? ;)

[j_scheibel]

@wellenreiter lol well i did quit my job just to work full time on my ideas, (so yeah, needed more time!). hopefully at least 1 becomes a good moneymaker or in the next couple years or i’ll have to go back to a day job!

as for the game and ftc payments… first, i was actually entertaining the idea to make the securities miner site’s paywall be in ftc (and only ftc) and usd. with like a 25% discount in price for ftc. (there is no paywall… not till traffic justifies it) anyway it’d be to see if anyone does it/uses it. i’d love to see it get some regular use.

I’ve looked in to integration,… so far i haven’t liked the amount of work it takes. I really want it to be as easy as install wallet and use wallet api’s to talk to the wallet, but somehow that seems risky. btcpay seems like A LOT of setup. so for now i’ve put it aside.

As for the video game. i’ll definitely think about it, and it’s a long way away… so by then i’ll know what’s what. my big concern is, by then (as it’s a year away) i’ll ~need~ cash flow (due to personal reserves running out) and if ftc’s market is strong then i can consider it, if it’s not then i’ll have to stick with paypal since i can’t risk utility/mortgatge payments on selling at a good price on an exchange.

Also, i’m still not a 100% sure if i’m doing this solo or on steam or some other game network. those other networks though probably wont let me do anything other than their integrated payment system. it’s definitely on my radar though. We’ll see how it goes.