I am self taught, formal education in a different field. I don't know if this would answer your question perfectly but I believe whatever success I had in delivering technology came from curiosity. I remember always thinking about how things run under the hood. Always ask questions. Pseudo-prototype how I would deliver it myself. This would shed a knowledge gap in certain areas because I would not be able to plan a certain step/process. Then I'd focus on that area and learn. It all starts with a hello world.
I've actually used OP's system in way higher traffic environments and was happy to read I was not alone. Sometimes all you need are simple tools and if git pull/ssh covers your basis, what's the big deal.
How actual software delivery is done is a relative term to the systems/needs and the scale you have. You probably have not been in environments where build/deploy tools are over engineered to death, to the point where adjusting it one bit brings down the entire delivery to a halt. If you were able to set those up without such issues, kudos to you.
I think what happens is some engineers invest so much of their professional identity in their Docker/K8s/Jenkins/AWS lifestyle that anything less is seen as dirty or inferior. The irony is that many of them are too young to remember the rationale for these tools when they were released, namely the management of massive fleets of servers by huge corporations. You see the same thing with the inappropriate use of React in the most minimally dynamic websites.
If you can apply the scraping to a use case and sell that use case you have a better chance at a viable business model. Examples come to my mind is; builtwith (scrape sites and publish the list of technologies they use), ahrefs (scrape sites and find outgoing/incoming links) etc.
Software engineering is one of the easiest jobs out there from an ability to track your output standpoint. I think a lot of the commenters implying "if you have 10 hours to cut down a tree, use the 9 to sharpen your ax" and using that to argue against code-line-output != productivity. Certainly in cases where you need to put a lot of thinking or research to come up with a very elegant solution in a few lines, no one can argue you have been productive. But more often than not, we work on simple CRUD or basic apps that were stretched among hours, flying under radar from one stand up to the next, like you mentioned.
It's a common problem so you are not the only one. Especially in large teams, or teams that are run by PM's who are not very savvy to open the hood and understand what had been going on. In most of the opposite cases, it's also awkward for some managers to challenge what you have produced in a day. It starts more on the foot of discovering: is there another mental block with the employee, are they having motivation issues (maybe you fall into this group), is there a training/onboarding issue? And I have seen first hand, most of the time - they give up and mediocre output continues.
Laziness/talent/ethics of it is not our place to comment for you - but it sounds like this would be a boring job to continue. Sure you can make good money and live a comfortable life, but I can't fathom if it would be fulfilling in the long run for me. Find what stirs your passion and try to apply to work. You have the luxury to do so.
I've briefly worked on an idea that was "pandora for news" when pandora was the hot music app. The app would learn from your votes and cater news/aggregations based on your preferences every day to your news dashboard. I was in love with the idea but never pursued it. Is there something like this now?
call this a romanticized view, but for every tool mankind has created - there are ways to utilize it in good VS bad outcomes. I didn't really think of it as; the learning would concentrate around opposing views. Instead of "liberal views on middle east" vs "conservative views on middle east" it would be about "middle east politics" whether you are interested in the topic or not in general - if that makes sense.
I would use a tool like this. That being said, that is not always a good indicator for a broad adoption interest.
> for every tool mankind has created - there are ways to utilize it in good VS bad outcomes
Counter point: Slot machines exist. (I suppose you might find some few instances of them being used to raise money for charity or something, but I don't think that redeems the technology.)
Recommendation engines are generally benign when the input content is benign. Pandora itself doesn't seem to cause trouble. Though even with benign content, content recommendation can encourage unhealthy behavior, such the content recommendation engine of netflix/etc encouraging binge consumption and couch potatoism.
But what does it mean for content to be benign and how is that determination made? When you have relatively little content that changes relatively infrequently, like netflix, you can have trusted humans perform the selection. That bakes the biases of those humans into the system, which is not inherently a problem (it's not as though any newspaper or book was ever any different in this regard.) But with things like youtube or the hypothetical Pandora for News, the the depth and breadth of content is too large for trusted human curation. I think we've actually already seen some examples of this manifest on Facebook, where the recommendation engine sends people down radicalizing holes of local news particularly.
There has been a lot of talk recently about the identification and suppression of disinformation and misinformation. D&M is problematic and tasking independent fact checkers to fight it may have some merit, but I think there is much more to consider. For starters, the local fact-checkers, who understand the language and cultural context of the material they are evaluating, are imposing their own potentially harmful biases onto the system. Secondly, and more importantly, some propaganda is neither misinformation nor disinformation. An example: racists love to pass around links to factually correct local news articles about crime. They will inundate their targets with news stories that support their narrative. Each of those stories, evaluated independently, might be factual and come from a reputable news organization. But the sum of those stories may mislead people by not presenting mitigating considerations, social or historical context, etc. If you create a system that recommends people the intersection of 'local news' and 'crime', you've automated the job of the racist propagandists. But if your content recommendation engine doesn't recommend such intersections, by lumping all local news into a single category, then it has little utility over simply visiting the websites of local news organizations. And if your system forbids local news entirely because the breadth of local news across the world is too much to moderate, then your system omits the news that directly impacts the lives of news readers the most.
I don't see any way around the above, but what makes a recommendation engine any worse than the local newspaper itself? Simply this: a newspaper is the same for everybody who reads it, when somebody is being sent down some bizarre radical rabbit-hole, those around them can read the newspaper and see what sort of thing their friend is reading. But recommendation engines provide personalized experiences. People can be radicalized without others in their community seeing what is happening, denying that community the opportunity to effectively respond and intervene. People withdraw into their own personal realities, losing touch with those around them. The further apart people grow, the harder reconciliation becomes.
I am willing to bet, attorneys who specialize in exerting money from ecommerce sites based on fluke accessibility lawsuits will not take notice of this problem that presents an actual hardship.
There are hundreds of ready made libraries which help you easily talk to SMTP without invoking a shell cmd. You also have the option to use sock functions (fsockopen) and write your connection wrapper. These are not windows-only features.
Of course. The part I still can't get my head around is where the language feature that'd relieve me of the need to vet a library, or worse write socket code by hand like some kind of barbarian, isn't compiled in for non-Windows platforms, because reasons.
If there is an application level function that alters the data that is stored before presenting to the client on the standard REST model, how is this case handled? Is it handled on the client side as well?
