tedshd's DevNote

Develop & Design Note by Ted

Chrome headless 研究筆記

Chrome headless 研究筆記

最近要用到 chrome headless 的部分功能

且得在 ubuntu or debian server 實踐

所以筆記一下

Intro

headless 大致分兩種用法

  1. CLI

  2. libary

Refer - Getting Started with Headless Chrome

CLI 用法簡單但是受限很多且沒有找到具體詳細完整的文件, 就連用 man 查都查不到...

也有可能是我自己沒有認真找

libary 就相對容易, 連 Google 自己都有提供部分語言的版本, 且有前端工程師很熟的 Node 版本

on ubuntu server

sudo apt-get install chromium-browser

ex:

chromium-browser --headless --screenshot=test.png --disable-gpu --window-size=320,480 --user-agent="Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) Version/7.0 Mobile/11D257 Safari/9537.53" http://tedshd.logdown.com

這指令下了截圖, 設定寬高, 模擬 UserAgent

當然也有 --dump-dom 可以看 source code

使用 puppeteer 處理

puppeteer

puppeteer 是 Google 維護的用 node 處理 headless 的套件

用了一下真是感到清爽好用

放一段最簡單的 smaple code

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    page.setUserAgent('Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:64.0) Gecko/20100101 Firefox/64.0');
        await page.goto('https://tysh310246.blogspot.com');

    await page.screenshot({path: 'example.png', fullPage: true});
    await browser.close();

})();

就是如此清爽

且有很多包好的 function 可以用

看了一下文件根本可以用 headless 取代 selenium + webdriver.io 了

但是目前的需求也不需要用到太多其他的功能

有空再慢慢研究

Puppeteer API Tip-Of-Tree

字型

在一台 server 上除非要截的網站是用 webfont 不然就會發現截的圖因為沒有字型所以中文會是方塊字

這部分就得處理一下

可以直接裝 Google 的 noto 字型

wget -c https://noto-website.storage.googleapis.com/pkgs/Noto-hinted.zip

把字型檔載下來放在

/usr/local/share/fonts/

設定 owner

設定權限

安裝字型

sudo fc-cache -fv

搞定

這邊建議直接裝全部語系的字型較為完整, 以免會遇到其他語系還是會有問題

Refer - Ubuntu環境下,手動安裝思源字型

2018 Google cloud 訓練營 筆記

2018 Google cloud 訓練營 筆記

Become a Google Cloud Platform expert with hands-on training. GCP Essentials

Gcloud cli

PROJECT_ID:

gcloud auth list

gcloud config list project

BASIC

gcloud -h

gcloud config --help || gcloud help config

gcloud config list --all

Managing Cloud Storage data

Try creating a Cloud Storage bucket. Bucket names must be unique, so replace unique-name with something else, or append the name to make it unique.

gsutil mb gs://<unique-name>

Upload file to bucket

gsutil cp <file> gs://<unique-name>

Create a new persistent disk

Because we want to attach this disk to the virtual machine instance we created in the previous step, the zone must be the same.

gcloud compute disks create <mydisk name> --size=200GB \ --zone us-central1-c

Attaching a disk

Attaching the persistent disk

You can attach a disk to a running virtual machine. Let's attach the new disk (mydisk) to the virtual machine instance you just created (gcelab).

gcloud compute instances attach-disk --disk --zone us-central1-c

Finding the persistent disk in the virtual machine

The persistent disk is now available as a block device in the virtual machine instance. Let's take a look.

go to this instance

ls -l /dev/disk/by-id/

You found the file, the default name is:

scsi-0Google_PersistentDisk_persistent-disk-1.

If you want a different device name, when you attach the disk, you would specify the device-name parameter. For example, to specify a device name, when you attach the disk you would use the command:

gcloud compute instances attach-disk gcelab --disk mydisk --device-name <YOUR_DEVICE_NAME> --zone us-central1-c

Formatting and mounting the persistent disk

Once you find the block device, you can partition the disk, format it, and then mount it using the following Linux utilities:

  • mkfs: creates a filesystem
  • mount: attaches to a filesystem

Make a mount point:

sudo mkdir /mnt/mydisk

Next, format the disk with a single ext4 filesystem using the mkfs tool. This command deletes all data from the specified disk:

sudo mkfs.ext4 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/disk/by-id/scsi-0Google_PersistentDisk_persistent-disk-1

Last lines of the output.

Allocating group tables: done
Writing inode tables: done
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done

Now use the mount tool to mount the disk to the instance with the discard option enabled:

sudo mount -o discard,defaults /dev/disk/by-id/scsi-0Google_PersistentDisk_persistent-disk-1 /mnt/mydisk

Q: Can you prevent the destruction of an attached persistent disk when the instance is deleted?

A: Yes, deselect the option Delete boot disk when instance is deleted when creating an instance

A:Yes, use the –keep-disks option with the gcloud compute instances delete command

Q: For migrating data from a persistent disk to another region, reorder the following steps in which they should be performed:

  1. Unmount file system(s)
  2. Create snapshot
  3. Create disk
  4. Create instance
  5. Attach disk

Hello Node Kubernetes

push docker image to instance

gcloud docker -- push gcr.io/<PROJECT_ID>/<hello-node>:<v1>

Create your cluster

gcloud config set project <PROJECT_ID>

Create a cluster with two n1-standard-1 nodes (this will take a few minutes to complete):

gcloud container clusters create <cluster name> \
                --num-nodes 2 \
                --machine-type n1-standard-1 \
                --zone us-central1-a

Note: You can also create this cluster through the Console, image shown above: Kubernetes Engine > Kubernetes clusters > Create cluster.

It is recommended to create the cluster in the same zone as the storage bucket used by the container registry (see previous step).

Create your pod

A Kubernetes pod is a group of containers tied together for administration and networking purposes. It can contain single or multiple containers. Here you'll use one container built with your Node.js image stored in your private container registry. It will serve content on port 8080.

Create a pod with the kubectl run command (replace PROJECT_ID with your GCP Project ID, found in the console and in the Connection Details section of the lab):

kubectl run <hello-node> \
    --image=gcr.io/<PROJECT_ID>/<hello-node>:v1 \
    --port=8080

hello-node is container image name

To view the deployment, run:

kubectl get deployments

To view the pod created by the deployment, run:

kubectl get pods
kubectl cluster-info

Overview of kubectl

And for troubleshooting :

kubectl get events

kubectl logs <pod-name>

Allow external traffic

By default, the pod is only accessible by its internal IP within the cluster. In order to make the hello-node container accessible from outside the Kubernetes virtual network, you have to expose the pod as a Kubernetes service.(預設只有內部 IP 可以讀取)

From Cloud Shell you can expose the pod to the public internet with the kubectl expose command combined with the --type="LoadBalancer" flag. This flag is required for the creation of an externally accessible IP:

kubectl expose deployment <container image name> --type="LoadBalancer"

To find the publicly-accessible IP address of the service, request kubectl to list all the cluster services:

kubectl get services

This is the output you should see:

NAME         CLUSTER-IP     EXTERNAL-IP      PORT(S)    AGE
hello-node   10.3.250.149   104.154.90.147   8080/TCP   1m
kubernetes   10.3.240.1     <none>           443/TCP    5m

open 104.154.90.147:8080

Scale up your service

One of the powerful features offered by Kubernetes is how easy it is to scale your application. Suppose you suddenly need more capacity for your application. You can tell the replication controller to manage a new number of replicas for your pod:

kubectl scale deployment <hello-node> --replicas=4

You can request a description of the updated deployment:

kubectl get deployment

You can also list the all pods:

kubectl get pods

Roll out an upgrade to your service

At some point the application that you've deployed to production will require bug fixes or additional features. Kubernetes helps you deploy a new version to production without impacting your users.

Modify files & rebuild & push

docker build -t gcr.io/<PROJECT_ID>/<hello-node>:v2 .
gcloud docker -- push gcr.io/<PROJECT_ID>/<hello-node>:v2

New version image

To do this, use the kubectl edit command. It opens a text editor displaying the full deployment yaml configuration. It isn't necessary to understand the full yaml config right now, just understand that by updating the spec.template.spec.containers.image field in the config you are telling the deployment to update the pods with the new image.

kubectl edit deployment <hello-node>

Modify image & save

Run the following to update the deployment with the new image. New pods will be created with the new image and the old pods will be deleted.

kubectl get deployments

While this is happening, the users of your services shouldn't see any interruption. After a little while they'll start accessing the new version of your application. You can find more details on rolling updates in this documentation.

Hopefully with these deployment, scaling, and updated features, once you've set up your Kubernetes Engine cluster, you'll agree that Kubernetes will help you focus on the application rather than the infrastructure.

Set Up Network and HTTP Load Balancers

In this hands-on lab, you'll learn the differences between a network load balancer and a HTTP load balancer, and how to set them up for your applications running on Google Compute Engine virtual machines.

There are several ways you can load balance in Google Cloud Platform. This lab takes you through the setup of the following load balancers.:

Set the default region and zone for all resources

In Cloud Shell, set the default zone:

gcloud config set compute/zone us-central1-a

Set the default region:

gcloud config set compute/region us-central1

Regions & Zones documentation

Create multiple web server instances

To simulate serving from a cluster of machines, create a simple cluster of Nginx web servers to serve static content using Instance Templates and Managed Instance Groups. Instance Templates define the look of every virtual machine in the cluster (disk, CPUs, memory, etc). Managed Instance Groups instantiate a number of virtual machine instances using the Instance Template.

To create the Nginx web server clusters, create the following:

  • A startup script to be used by every virtual machine instance to setup Nginx server upon startup
  • An instance template to use the startup script
  • A target pool
  • A managed instance group using the instance template

Still in Cloud Shell, create a startup script to be used by every virtual machine instance. This script sets up the Nginx server upon startup:

cat << EOF > startup.sh
#! /bin/bash
apt-get update
apt-get install -y nginx
service nginx start
sed -i -- 's/nginx/Google Cloud Platform - '"\$HOSTNAME"'/' /var/www/html/index.nginx-debian.html
EOF

Create an instance template, which uses the startup script:

gcloud compute instance-templates create nginx-template \
         --metadata-from-file startup-script=startup.sh

Create a target pool. A target pool allows a single access point to all the instances in a group and is necessary for load balancing in the future steps.

gcloud compute target-pools create nginx-pool

Create a managed instance group using the instance template:

gcloud compute instance-groups managed create nginx-group \
         --base-instance-name nginx \
         --size 2 \
         --template nginx-template \
         --target-pool nginx-pool \

List instance

gcloud compute instances list

Now configure a firewall so that you can connect to the machines on port 80 via the EXTERNAL_IP addresses:

gcloud compute firewall-rules create www-firewall --allow tcp:80

Create a Network Load Balancer

Network load balancing allows you to balance the load of your systems based on incoming IP protocol data, such as address, port, and protocol type. You also get some options that are not available, with HTTP(S) load balancing. For example, you can load balance additional TCP/UDP-based protocols such as SMTP traffic. And if your application is interested in TCP-connection-related characteristics, network load balancing allows your app to inspect the packets, where HTTP(S) load balancing does not.

For more information, see Setting Up Network Load Balancing.

Create an L3 network load balancer targeting your instance group:

gcloud compute forwarding-rules create nginx-lb \
         --region us-central1 \
         --ports=80 \
         --target-pool nginx-pool

List all Google Compute Engine forwarding rules in your project.

gcloud compute forwarding-rules list

Create a HTTP(s) Load Balancer

HTTP(S) load balancing provides global load balancing for HTTP(S) requests destined for your instances. You can configure URL rules that route some URLs to one set of instances and route other URLs to other instances. Requests are always routed to the instance group that is closest to the user, provided that group has enough capacity and is appropriate for the request. If the closest group does not have enough capacity, the request is sent to the closest group that does have capacity.

Learn more about the HTTP(s) Load Balancer in the documentation.

First, create a health check. Health checks verify that the instance is responding to HTTP or HTTPS traffic:

gcloud compute http-health-checks create http-basic-check

Define an HTTP service and map a port name to the relevant port for the instance group. Now the load balancing service can forward traffic to the named port:

gcloud compute instance-groups managed \
       set-named-ports nginx-group \
       --named-ports http:80

Create a backend service:

gcloud compute backend-services create nginx-backend \
      --protocol HTTP --http-health-checks http-basic-check --global

Add the instance group into the backend service:

gcloud compute backend-services add-backend nginx-backend \
    --instance-group nginx-group \
    --instance-group-zone us-central1-a \
    --global

Create a default URL map that directs all incoming requests to all your instances:

gcloud compute url-maps create web-map \
    --default-service nginx-backend

To direct traffic to different instances based on the URL being requested, see content-based routing.

Create a target HTTP proxy to route requests to your URL map:

gcloud compute target-http-proxies create http-lb-proxy \
    --url-map web-map

Create a global forwarding rule to handle and route incoming requests. A forwarding rule sends traffic to a specific target HTTP or HTTPS proxy depending on the IP address, IP protocol, and port specified. The global forwarding rule does not support multiple ports.

gcloud compute forwarding-rules create http-content-rule \
        --global \
        --target-http-proxy http-lb-proxy \
        --ports 80

After creating the global forwarding rule, it can take several minutes for your configuration to propagate.

gcloud compute forwarding-rules list

Finally

GCP Essentials

JavaScript - 計算網頁元件的曝光

JavaScript - 計算網頁元件的曝光

因為某些需求需要精確計算某些網頁上的 UI 元件的曝光

沒錯

就是要像廣告計算 impression 一樣

之前只要確認有該元件的 request 就好了

但是為了精確追求計算曝光, 所以必須做到像 Google Ad 一樣

要到該網頁元件讓使用者看到後才能計算曝光

聽起來這要求不太容易

但是實作下去其實發現還好誒!!

概念

DOM 元件進入可視範圍觸發 impression

所以需要能夠釐清一些問題

  1. 如何確認可視範圍?

  2. 如何確認要記錄 impression 的 DOM 元件在可視範圍?

  3. 如何在進入可視範圍時觸發?

看似困難

但是如果熟悉 JavaScript Window 物件和 DOM 物件的話就會發現很好解

實作

如何確認可視範圍?
window.document.documentElement.clientHeight

window.document.documentElement.clientWidth

完美解決

如何確認要記錄 impression 的 DOM 元件在可視範圍?
<DOM Object>.getBoundingClientRect();

完美解決

但是要取到對應可是範圍的位置建議使用(top, left), x y IE 和 Edge 不支援

還有一點要注意就是該 DOM 要先長在 browser 上面且不是 display: none 的狀態才能拿到值

Refer - Element.getBoundingClientRect() - Web API 接口 | MDN

Refer - ClientRect object (Internet Explorer)

如何在進入可視範圍時觸發?

這就得取決於使用情境了

在適當的 Event 去觸發

Ex:

在 list view 時就單純的用 scroll 事件觸發確認是否是進入可視範圍(當然得做 debounce 處理)

在 carousel view 就滑動或點擊觸發轉動時確認是否是進入可視範圍(如果有做動畫效果, 得等到動畫效果結束再去確認)

結論

熟悉基本 API 後就可以很輕鬆地解決很多問題 XD

最後附上包好的套件

GitHub - impression

LINE - 2018 1111 分享得 LINE point 研究筆記與分析報告

LINE - 2018 1111 分享得 LINE point 研究筆記與分析報告

MGM(member get member) 的模式一直以來都會是一個有規則可循且有時難度也不高的活動模式(因為難度高就不好擴散

這次以 LINE 2018 年 1111 分享連結的 LINE point 為例子當作我們這次的研究對象

大致規則

分享該訊息給朋友

會是如下圖所示

活動辦法引言

活動辦法:
● 好友點選您分享的連結進入LINE購物首頁,若成功您將會收到『LINE購物官方帳號』通知抽獎囉。分享者需透過活動頁分享網址,邀請朋友點入LINE購物並完成登入,才能獲得點數。
● 轉發他人的分享網址,點數將會計算給原分享者。
● 請先加入『LINE購物官方帳號』好友以及解除封鎖可獲得參與活動資格與收到LINE 購物抽獎/得獎通知。
● 分享給同一名好友,最高可獲得1次抽獎機會,若要再獲得抽獎機會請分享給其他好友。
● 各平台折扣碼「LINE11」,使用期間:2018/11/10 - 2018/11/11折抵金額與使用方式請見各平台規範。活動詳請請見:https://lin.ee/5JVllUj/rcf
● 此分享活動時間為2018/11/5 11:00至2018/11/9 23:59(台灣時間)。收到抽獎券之用戶,須於2018/11/12 23:59(台灣時間)前點擊抽獎網址,逾期未完成抽獎者視同放棄並自動取消資格。
● 本次活動限LINE帳號已綁定台灣手機門號用戶參加

那上述規則就令人產生一些聯想

那大家互相分享再去點開連結就可以得到點數了呢!

研究

本持著攻城獅的研究精神

就發現有人分享了一份 Google 表單

上面記錄了許多人填出來的分享 URL(當然也有來亂的XD)

那麼接下來就可以開始一件事, 那就是點連結拉

但是在 Line 開這表單再點連結好累喔

那這時候又有個思路就是可以讓這過程自動化阿

但是要在模擬器上登入 LINE 很麻煩(因為 LINE 只能同時登入一台手機)

那換個思路

既然是連結那總可以在 browser 打開吧

ok

當你那麼做以後又發現一件事了

好像沒用誒

當然你把分享連結在電腦上用 browser 開

他怎麼知道是哪一個 user 打開的呢

那是否能解決這問題呢?

是的

可以解決

拜 LINE 越來越開放的機制還有 LINE APP 大量使用 webview 的關係

所以現在可以在電腦 browser 登入 LINE 帳號(其實主要是開發 LINE bot 或使用 LINE API 等服務使用的)

那麼這問題就迎刃而解了

身為一個攻城獅還是略懂略懂怎麼讓這個東西自動化的

這裡分享個其他人的做法

有一些方法可以大量開啟URL

但是會是一次大量開啟

所以說如果有 3000 個連結, 一次開 3000 個連結電腦不當掉才怪

實作

所以只要搞一個方式是開一個分頁然後一直把連結在那個分頁開啟即可

但是身為一個注重使用者體驗的攻城獅

一定要有良好的使用體驗和輕鬆入門的方式

目標一定是要用最簡當的方式讓不懂程式的廣大民眾能輕鬆入門

所以就是提供一段程式碼請一般人貼到 chrome 的開發者工具中的 console 即可

  1. 方便使用

  2. 可以設定多久開下一個連結

  3. 會用 localstorage 紀錄上次開到哪個連結, 下次就可以繼續不必重跑

大致就可以達到目的了

但是得到點數的通知還是要去 LINE 裡面點試試手氣誒!

沒有太大的問題

那麼整個活動到了最後一步

抽獎

LINE 會以通知的方式通知 user 可以點 LINE 裡的 show card 去抽獎

那麼只要有辦法拿到點抽獎頁的 link 即可(因為那頁是 WebView XD)

那頁特別的是使用 line: 這個自訂的 protocol 所以得在 LINE 裡面打開

那麼你把連結貼在 LINE 的訊息上即可在那一直點了

不用去 LINE 購物裡面點

那麼最後一步就是寫自動化去自動抽點數了拉(不用再自己點了 XD

但是這有個門檻在

因為手機自動化測試的環境安裝有一點難度

但網路上也有許多自動化測試的文章這裡就不詳述了

起源

小弟原本興致缺缺

但是看到第一天小弟所在的群組有人在討論

然後表單就蹦的出來了

然後就蹦的出現上面開大量連結的方法

然後小弟表示哪有那麼傻逼的方法

電腦哪撐得住阿, 而且表單會一直更新, 那麼傻逼的行為我無法接受

所以就提供了上述的方法

原本想說搞完連結就算了

但是就不知不覺就陷進去了...

然後就搞到最後都全自動了...

活動觀察

其實小弟活動第二天才參與

所以第一天的高峰期就錯過了

因為第一天 LINE 沒有做足夠的準備

第一天都可以一直抽到點數, 但是抽點的頁面被 DDOS, 幾乎沒法進去

然後第二天 LINE 就把抽不中的機率提高到一個不可思議的地步

而且似乎把發放的 LINE point 數量做了數量控管

但是持續間隔一小時去打的話還是能試出來何時有點數可以抽

結語

做 MGM 活動真的要小心

連我們自己公司的服務在做的時候比 LINE 更嚴謹都還是會出現有人弄成程式去跑(我們還強制綁 SMS 一樣沒用, 對方似乎是用 提供手機號碼的服務去繞過這一段)

基本上因為自動化程式的出現, 只要花點時間了解規則和撰寫腳本, 是不容易防不住的

當然也是還有很多增加難度的方式

例如:

  1. 擋 IP

  2. 擋住該使用者

  3. 設定條件上限

這些都是增加難度的一些措施

當然還有很重要的是主辦方是有權利修改活動辦法抑或發現是有非正常流程與方式來完成活動可以取消資格的方式來處理